.. note::
- Scripts are stored in the OpenDaylight distribution/bin folder, and
+ Scripts are stored in the OpenDaylight ``distribution/bin`` folder, and
maintained in the distribution project
- `repository <https://git.opendaylight.org/gerrit/p/integration/distribution>`_
- in the folder distribution-karaf/src/main/assembly/bin/.
+ `repository <https://git.opendaylight.org/gerrit/admin/repos/integration/distribution>`_
+ in the folder ``karaf-scripts/src/main/assembly/bin/``.
Configure Cluster Script
^^^^^^^^^^^^^^^^^^^^^^^^
-This script is used to configure the cluster parameters (e.g. akka.conf,
-module-shards.conf) on a member of the controller cluster. The user should
+This script is used to configure the cluster parameters (e.g. ``akka.conf``,
+``module-shards.conf``) on a member of the controller cluster. The user should
restart the node to apply the changes.
.. note::
seed_node_list the same, and vary the index from 1 through N.
Optionally, shards can be configured in a more granular way by modifying the
-file "custom_shard_configs.txt" in the same folder as this tool. Please see
+file ``"custom_shard_configs.txt"`` in the same folder as this tool. Please see
that file for more details.
Example::
#. Copy the OpenDaylight distribution zip file to the machine.
#. Unzip the distribution.
-#. Open the following .conf files:
+#. Move into the ``<karaf-distribution-directory>/bin`` directory and run::
- * configuration/initial/akka.conf
- * configuration/initial/module-shards.conf
+ JAVA_MAX_MEM=4G JAVA_MAX_PERM_MEM=512m ./karaf
+
+#. Enable clustering by running the following command at the Karaf command line::
+
+ feature:install odl-mdsal-distributed-datastore
+
+ After installation you will be able to see new folder ``configuration/initial/``
+ with config files
+
+#. Open the following configuration files:
+
+ * ``configuration/initial/akka.conf``
+ * ``configuration/initial/module-shards.conf``
#. In each configuration file, make the following changes:
hostname or IP address of the machine on which this file resides and
OpenDaylight will run::
- netty.tcp {
- hostname = "127.0.0.1"
+ artery {
+ canonical.hostname = "127.0.0.1"
.. note:: The value you need to specify will be different for each node in the
cluster.
address of any of the machines that will be part of the cluster::
cluster {
- seed-nodes = ["akka.tcp://opendaylight-cluster-data@${IP_OF_MEMBER1}:2550",
+ seed-nodes = ["akka://opendaylight-cluster-data@${IP_OF_MEMBER1}:2550",
<url-to-cluster-member-2>,
<url-to-cluster-member-3>]
.. note:: This step should use a different role on each node.
-#. Open the configuration/initial/module-shards.conf file and update the
+#. Open the ``configuration/initial/module-shards.conf`` file and update the
replicas so that each shard is replicated to all three nodes::
replicas = [
"member-3"
]
- For reference, view a sample config files <<_sample_config_files,below>>.
-
-#. Move into the +<karaf-distribution-directory>/bin+ directory.
-#. Run the following command::
-
- JAVA_MAX_MEM=4G JAVA_MAX_PERM_MEM=512m ./karaf
+ For reference, view a sample config files below.
-#. Enable clustering by running the following command at the Karaf command line::
+#. Restart bundle via command line::
- feature:install odl-mdsal-clustering
+ opendaylight-user@root>restart org.opendaylight.controller.sal-distributed-datastore
OpenDaylight should now be running in a three node cluster. You can use any of
the three member nodes to access the data residing in the datastore.
Sample ``akka.conf`` file::
odl-cluster-data {
- bounded-mailbox {
- mailbox-type = "org.opendaylight.controller.cluster.common.actor.MeteredBoundedMailbox"
- mailbox-capacity = 1000
- mailbox-push-timeout-time = 100ms
- }
-
- metric-capture-enabled = true
-
akka {
- loglevel = "DEBUG"
- loggers = ["akka.event.slf4j.Slf4jLogger"]
-
- actor {
-
- provider = "akka.cluster.ClusterActorRefProvider"
- serializers {
- java = "akka.serialization.JavaSerializer"
- proto = "akka.remote.serialization.ProtobufSerializer"
- }
-
- serialization-bindings {
- "com.google.protobuf.Message" = proto
-
- }
- }
remote {
- log-remote-lifecycle-events = off
- netty.tcp {
- hostname = "10.194.189.96"
- port = 2550
- maximum-frame-size = 419430400
- send-buffer-size = 52428800
- receive-buffer-size = 52428800
+ artery {
+ enabled = on
+ transport = tcp
+ canonical.hostname = "10.0.2.10"
+ canonical.port = 2550
}
}
cluster {
- seed-nodes = ["akka.tcp://opendaylight-cluster-data@10.194.189.96:2550",
- "akka.tcp://opendaylight-cluster-data@10.194.189.98:2550",
- "akka.tcp://opendaylight-cluster-data@10.194.189.101:2550"]
-
- auto-down-unreachable-after = 10s
+ # Using artery.
+ seed-nodes = ["akka://opendaylight-cluster-data@10.0.2.10:2550",
+ "akka://opendaylight-cluster-data@10.0.2.11:2550",
+ "akka://opendaylight-cluster-data@10.0.2.12:2550"]
roles = [
- "member-2"
+ "member-1"
]
+ # when under load we might trip a false positive on the failure detector
+ # failure-detector {
+ # heartbeat-interval = 4 s
+ # acceptable-heartbeat-pause = 16s
+ # }
}
- }
- }
- odl-cluster-rpc {
- bounded-mailbox {
- mailbox-type = "org.opendaylight.controller.cluster.common.actor.MeteredBoundedMailbox"
- mailbox-capacity = 1000
- mailbox-push-timeout-time = 100ms
- }
+ persistence {
+ # By default the snapshots/journal directories live in KARAF_HOME. You can choose to put it somewhere else by
+ # modifying the following two properties. The directory location specified may be a relative or absolute path.
+ # The relative path is always relative to KARAF_HOME.
- metric-capture-enabled = true
-
- akka {
- loglevel = "INFO"
- loggers = ["akka.event.slf4j.Slf4jLogger"]
-
- actor {
- provider = "akka.cluster.ClusterActorRefProvider"
+ # snapshot-store.local.dir = "target/snapshots"
+ # Use lz4 compression for LocalSnapshotStore snapshots
+ snapshot-store.local.use-lz4-compression = false
+ # Size of blocks for lz4 compression: 64KB, 256KB, 1MB or 4MB
+ snapshot-store.local.lz4-blocksize = 256KB
}
- remote {
- log-remote-lifecycle-events = off
- netty.tcp {
- hostname = "10.194.189.96"
- port = 2551
- }
- }
-
- cluster {
- seed-nodes = ["akka.tcp://opendaylight-cluster-rpc@10.194.189.96:2551"]
-
- auto-down-unreachable-after = 10s
- }
+ disable-default-actor-system-quarantined-event-handling = "false"
}
}
Cluster Monitoring
------------------
-OpenDaylight exposes shard information via MBeans, which can be explored with
-JConsole, VisualVM, or other JMX clients, or exposed via a REST API using
-`Jolokia <https://jolokia.org/features-nb.html>`_, provided by the
+OpenDaylight exposes shard information via ``MBeans``, which can be explored
+with ``JConsole``, VisualVM, or other JMX clients, or exposed via a REST API using
+`Jolokia <https://jolokia.org/features.html>`_, provided by the
``odl-jolokia`` Karaf feature. This is convenient, due to a significant focus
on REST in OpenDaylight.
-The basic URI that lists a schema of all available MBeans, but not their
+The basic URI that lists a schema of all available ``MBeans``, but not their
content itself is::
GET /jolokia/list
"LastApplied": 5,
"LastLeadershipChangeTime": "2017-01-06 13:18:37.605",
"LastLogIndex": 5,
- "PeerAddresses": "member-3-shard-default-operational: akka.tcp://opendaylight-cluster-data@192.168.16.3:2550/user/shardmanager-operational/member-3-shard-default-operational, member-2-shard-default-operational: akka.tcp://opendaylight-cluster-data@192.168.16.2:2550/user/shardmanager-operational/member-2-shard-default-operational",
+ "PeerAddresses": "member-3-shard-default-operational: akka://opendaylight-cluster-data@192.168.16.3:2550/user/shardmanager-operational/member-3-shard-default-operational, member-2-shard-default-operational: akka://opendaylight-cluster-data@192.168.16.2:2550/user/shardmanager-operational/member-2-shard-default-operational",
"WriteOnlyTransactionCount": 0,
"FollowerInitialSyncStatus": false,
"FollowerInfo": [
The ODLTools team is maintaining a Python based `tool
<https://github.com/opendaylight/odltools>`_,
-that takes advantage of the above MBeans exposed via Jolokia.
+that takes advantage of the above ``MBeans`` exposed via ``Jolokia``.
.. _cluster_admin_api:
This strategy is used by default, because it works well for most systems.
It will down the unreachable nodes if the current node is in the majority part
-based on the last known membership information. Otherwise down the reachable nodes,
-i.e. the own part. If the parts are of equal size the part containing the node with
-the lowest address is kept.
+based on the last known membership information. Otherwise down the reachable
+nodes, i.e. the own part. If the parts are of equal size the part containing the
+node with the lowest address is kept.
This strategy is a good choice when the number of nodes in the cluster change
dynamically and you can therefore not use static-quorum.
cluster, or when you can define a fixed number of nodes with a certain role.
* If there are unreachable nodes when starting up the cluster, before reaching
- this limit, the cluster may shut itself down immediately. This is not an issue
- if you start all nodes at approximately the same time or use the
- akka.cluster.min-nr-of-members to define required number of members before the
- leader changes member status of ‘Joining’ members to ‘Up’. You can tune the
- timeout after which downing decisions are made using the stable-after setting.
+ this limit, the cluster may shut itself down immediately.
+ This is not an issue if you start all nodes at approximately the same time or
+ use the ``akka.cluster.min-nr-of-members`` to define required number of
+ members before the leader changes member status of ‘Joining’ members to ‘Up’.
+ You can tune the timeout after which downing decisions are made using the
+ stable-after setting.
* You should not add more members to the cluster than quorum-size * 2 - 1.
- If the exceeded cluster size remains when a SBR decision is needed it will down
- all nodes because otherwise there is a risk that both sides may down each other
- and thereby form two separate clusters.
+ If the exceeded cluster size remains when a SBR decision is needed it will
+ down all nodes because otherwise there is a risk that both sides may down each
+ other and thereby form two separate clusters.
* If the cluster is split into 3 (or more) parts each part that is smaller than
- then configured quorum-size will down itself and possibly shutdown the whole cluster.
+ then configured quorum-size will down itself and possibly shutdown the whole
+ cluster.
* If more nodes than the configured quorum-size crash at the same time the other
running nodes will down themselves because they think that they are not in the
if the oldest was Leaving and not changed to Exiting then each part will shut down
itself, terminating the whole cluster.
-The decision can be based on nodes with a configured role instead of all nodes in
-the cluster.
+The decision can be based on nodes with a configured role instead of all nodes
+in the cluster.
Configuration::
The strategy named lease-majority is using a distributed lease (lock) to decide what
nodes that are allowed to survive. Only one SBR instance can acquire the lease make
-the decision to remain up. The other side will not be able to aquire the lease and
+the decision to remain up. The other side will not be able to acquire the lease and
will therefore down itself.
This strategy is very safe since coordination is added by an external arbiter.
Indirectly connected nodes
^^^^^^^^^^^^^^^^^^^^^^^^^^
-In a malfunctional network there can be situations where nodes are observed as
+In a malfunctioning network there can be situations where nodes are observed as
unreachable via some network links but they are still indirectly connected via
other nodes, i.e. it’s not a clean network partition (or node crash).
max-shard-data-store-executor-queue-size uint32 (1..max) 5000 The maximum queue size for each shard's data store executor.
shard-transaction-idle-timeout-in-minutes uint32 (1..max) 10 The maximum amount of time a shard transaction can be idle without receiving any messages before it self-destructs.
shard-snapshot-batch-count uint32 (1..max) 20000 The minimum number of entries to be present in the in-memory journal log before a snapshot is to be taken.
-shard-snapshot-data-threshold-percentage uint8 (1..100) 12 The percentage of Runtime.totalMemory() used by the in-memory journal log before a snapshot is to be taken
-shard-hearbeat-interval-in-millis uint16 (100..max) 500 The interval at which a shard will send a heart beat message to its remote shard.
+shard-snapshot-data-threshold-percentage uint8 (1..100) 12 The percentage of ``Runtime.totalMemory()`` used by the in-memory journal log before a snapshot is to be taken
+shard-heartbeat-interval-in-millis uint16 (100..max) 500 The interval at which a shard will send a heart beat message to its remote shard.
operation-timeout-in-seconds uint16 (5..max) 5 The maximum amount of time for akka operations (remote or local) to complete before failing.
shard-journal-recovery-log-batch-size uint32 (1..max) 5000 The maximum number of journal log entries to batch on recovery for a shard before committing to the data store.
shard-transaction-commit-timeout-in-seconds uint32 (1..max) 30 The maximum amount of time a shard transaction three-phase commit can be idle without receiving the next messages before it aborts the transaction
shard-isolated-leader-check-interval-in-millis uint32 (1..max) 5000 the interval at which the leader of the shard will check if its majority followers are active and term itself as isolated
============================================== ================= ======= ==============================================================================================================================================================================
-These configuration options are included in the etc/org.opendaylight.controller.cluster.datastore.cfg configuration file.
+These configuration options are included in the ``etc/org.opendaylight.controller.cluster.datastore.cfg`` configuration file.
Code Review / docs.git / blobdiff