-== Setting Up Clustering on an OpenDaylight Controller
-
-//* <<Clustering Overview>>
-//* <<Single Node Clustering>>
-//* <<Multiple Node Clustering>>
-//** <<Deployment Considerations>>
-//** <<Setting Up a Multiple Node Cluster>>
-//*** <<Enabling HA on a Multiple Node Cluster>>
+== Setting Up Clustering
=== Clustering Overview
-Clustering is a mechanism that enables multiple processes and programs to work together as one entity. For example, when you go to google.com and search for something, it may seem like your search request is processed by only one web server. In reality, your search request is processed by thousands of web servers connected in a cluster. Similarly, you can have multiple instances of the OpenDaylight controller working together as one entity. There are a number of uses for clustering:
-
-* Scaling: If you have multiple controllers running, you can potentially do more work with or store more data on those controllers if they are clustered. You can also break up your data into smaller chunks (known as shards) and either distribute that data across the cluster or perform certain operations on certain members of the cluster.
-
-* High Availability: If you have multiple controllers running and one of them crashes, you would still have the other instances working and available.
-
-* Data Persistence: You will not lose any data gathered by your controller after a manual restart or a crash.
-
-The following sections describe how to set up clustering on both individual and multiple OpenDaylight controllers.
+Clustering is a mechanism that enables multiple processes and programs to work
+together as one entity. For example, when you search for something on
+google.com, it may seem like your search request is processed by only one web
+server. In reality, your search request is processed by may web servers
+connected in a cluster. Similarly, you can have multiple instances of
+OpenDaylight working together as one entity.
+
+Advantages of clustering are:
+
+* Scaling: If you have multiple instances of OpenDaylight running, you can
+ potentially do more work and store more data than you could with only one
+ instance. You can also break up your data into smaller chunks (shards) and
+ either distribute that data across the cluster or perform certain operations
+ on certain members of the cluster.
+* High Availability: If you have multiple instances of OpenDaylight running and
+ one of them crashes, you will still have the other instances working and
+ available.
+* Data Persistence: You will not lose any data stored in OpenDaylight after a
+ manual restart or a crash.
+
+The following sections describe how to set up clustering on both individual and
+multiple OpenDaylight instances.
=== Single Node Clustering
-To enable clustering on a single OpenDaylight controller, do the following:
-
-'''
-. Download and unzip a base controller distribution. You must use the new OpenFlow plugin, so download a distribution where the new OpenFlow plugin is either the default or can be enabled.
-. Navigate to the _<Karaf-distribution-location>_/bin directory.
-. Run Karaf: *./karaf*
-. Install the clustering feature: *feature:install odl-mdsal-clustering*
-. If you are using the integration distribution of Karaf, you should also install the open flow plugin flow services: *feature:install odl-openflowplugin-flow-services*
-. Install the Jolokia bundle: *install -s mvn:org.jolokia/jolokia-osgi/1.1.5*
+To enable clustering on a single instance of OpenDaylight, perform the
+following steps:
-'''
+. Download, install, and run OpenDaylight as
+ <<_getting_and_installing_opendaylight_lithium,described above>>
+. Install the clustering feature:
++
+ feature:install odl-mdsal-clustering
-After enabling the DistributedDataStore feature in a single instance, you can access the following features:
-
-* Data Sharding: The in-memory MD-SAL tree is broken up into a number of smaller sub-trees (inventory, topology, and default).
-* Data Persistence: All of the data available on defined data shards is stored on a disk. By restarting the controller, you can use the persisted data to reinstate those shards to their previous state.
+NOTE: This will enabled the cluster-ready version of the MD-SAL data store, but
+ will not actually create a cluster of multiple instances. The result is
+ that you will get data persistence, but not the scaling or high
+ availability advantages.
=== Multiple Node Clustering
==== Deployment Considerations
-Here is some information to keep in mind when you implement clustering:
-
-* When setting up a cluster with multiple nodes, we recommend that you do so with a minimum of three machines. You can set up with a cluster with just two nodes. However, if one of those two nodes go down, the controller will no longer be operational.
-
-* Every device that belongs to a cluster needs to have an identifier. For this purpose, OpenDaylight uses the node’s role. After you define the first node’s role as _member-1_ in the akka.conf file, OpenDaylight uses _member-1_ to identify that node.
-
-* Data shards are used to house all or a certain segment of a module’s data. For example, one shard can contain all of a module’s inventory data while another shard contains all of it’s topology data. If you do not specify a module in the modules.conf file and do not specify a shard in module-shards.conf, then (by default) all the data is places onto the default shard (which must also be defined in module-shards.conf file). Each shard has replicas configured, and the module-shards.conf file is where you can specify where these replicas reside.
-
-* Say you have a three node cluster on which HA is enabled. A replica of every defined data shard must be running on all three cluster nodes. This is because OpenDaylight’s clustering implementation requires a majority of the defined shard replicas to be running in order to function. If you only define data shard replicas on two of the cluster nodes and one of those nodes goes down, the corresponding data shards will not function.
-
-* If you have a three node cluster and have defined replicas for a data shard on each of those nodes, that shard will still function even if only two of the cluster nodes are running. Note that if one of those two nodes go down, your controller will no longer be operational.
-
-*What considerations need to be made when setting the seed nodes for each member? Why are we referring to multiple seed nodes when you set only one IP address? Can you set multiple seed nodes for functional testing?*
-
-We recommend that you have multiple seed nodes configured. After a cluster member is started, it sends a message to all of its seed nodes. The cluster member then sends a join command to the first seed node that responds. If none of its seed nodes reply, the cluster member repeats this process until it successfully establishes a connection or it is shutdown.
-
-*What happens after one node becomes unreachable? Do the other two nodes function normally? When the first node reconnects, does it automatically synchronize with the other nodes?*
-
-After a node becomes unreachable, it remains down for configurable period of time (10 seconds, by default). Once a node goes down, you need to restart it so that it can rejoin the cluster. Once a restarted node joins a cluster, it will synchronize with the lead node automatically.
-
-*Can you run a two node cluster for functional testing?*
-
-For functional testing, yes. For HA testing, you need to run all three nodes.
+To implement clustering, the deployment considerations are as follows:
+
+* To set up a cluster with multiple nodes, we recommend that you use a minimum
+ of three machines. You can set up a cluster with just two nodes. However, if
+ one of the two nodes fail, the cluster will not be operational.
++
+NOTE: This is because clustering in OpenDaylight requires a majority of the
+ nodes to be up and one node cannot be a majority of two nodes.
++
+* Every device that belongs to a cluster needs to have an identifier.
+ OpenDaylight uses the node's +role+ for this purpose. After you define the
+ first node's role as _member-1_ in the +akka.conf+ file, OpenDaylight uses
+ _member-1_ to identify that node.
+
+* Data shards are used to contain all or a certain segment of a OpenDaylight's
+ MD-SAL datastore. For example, one shard can contain all the inventory data
+ while another shard contains all of the topology data.
++
+If you do not specify a module in the +modules.conf+ file and do not specify
+a shard in +module-shards.conf+, then (by default) all the data is placed in
+the default shard (which must also be defined in +module-shards.conf+ file).
+Each shard has replicas configured. You can specify the details of where the
+replicas reside in +module-shards.conf+ file.
+
+* If you have a three node cluster and would like to be able to tolerate any
+ single node crashing, a replica of every defined data shard must be running
+ on all three cluster nodes.
++
+NOTE: This is because OpenDaylight's clustering implementation requires a
+ majority of the defined shard replicas to be running in order to
+ function. If you define data shard replicas on two of the cluster nodes
+ and one of those nodes goes down, the corresponding data shards will not
+ function.
++
+* If you have a three node cluster and have defined replicas for a data shard
+ on each of those nodes, that shard will still function even if only two of
+ the cluster nodes are running. Note that if one of those remaining two nodes
+ goes down, the shard will not be operational.
+
+* It is recommended that you have multiple seed nodes configured. After a
+ cluster member is started, it sends a message to all of its seed nodes.
+ The cluster member then sends a join command to the first seed node that
+ responds. If none of its seed nodes reply, the cluster member repeats this
+ process until it successfully establishes a connection or it is shut down.
+
+* After a node is unreachable, it remains down for configurable period of time
+ (10 seconds, by default). Once a node goes down, you need to restart it so
+ that it can rejoin the cluster. Once a restarted node joins a cluster, it
+ will synchronize with the lead node automatically.
==== Setting Up a Multiple Node Cluster
-To run an OpenDaylight controller in a three node cluster, do the following:
-
-'''
+To run OpenDaylight in a three node cluster, perform the following:
-. Determine the three machines that will make up the cluster and copy the controller distribution to each of those machines.
-. Unzip the controller distribution.
-. Navigate to the _<Karaf-distribution-location>_/bin directory.
-. Run Karaf: *./karaf*
-. Install the clustering feature: *feature:install odl-mdsal-clustering*
-
-NOTE: To run clustering, you must install the odl-mdsal-clustering feature on each of your nodes.
-
-[start=6]
-. If you are using the integration distribution of Karaf, you should also install the open flow plugin flow services: *feature:install odl-openflowplugin-flow-services*
-. Install the Jolokia bundle: *install -s mvn:org.jolokia/jolokia-osgi/1.1.5*
-. On each node, open the following .conf files:
-
-* configuration/initial/akka.conf
-* configuration/initial/module-shards.conf
+First, determine the three machines that will make up the cluster. After that,
+do the following on each machine:
+. Copy the OpenDaylight distribution zip file to the machine.
+. Unzip the distribution.
+. Open the following .conf files:
+** configuration/initial/akka.conf
+** configuration/initial/module-shards.conf
. In each configuration file, make the following changes:
-
-.. Find every instance of the following lines and replace _127.0.0.1_ with the hostname or IP address of the machine on which the controller will run:
-
+.. Find every instance of the following lines and replace _127.0.0.1_ with the
+ hostname or IP address of the machine on which this file resides and
+ OpenDaylight will run:
++
netty.tcp {
hostname = "127.0.0.1"
-
-NOTE: The value you need to specify will be different for each node in the cluster.
-
-[start=2]
-.. Find the following lines and replace _127.0.0.1_ with the hostname or IP address of any of the machines that will be part of the cluster:
-
++
+NOTE: The value you need to specify will be different for each node in the
+ cluster.
++
+.. Find the following lines and replace _127.0.0.1_ with the hostname or IP
+ address of any of the machines that will be part of the cluster:
++
cluster {
seed-nodes = ["akka.tcp://opendaylight-cluster-data@127.0.0.1:2550"]
-
-.. Find the following section and specify the role for each member node. For example, you could assign the first node with the _member-1_ role, the second node with the _member-2_ role, and the third node with the _member-3_ role.
-
-
++
+.. Find the following section and specify the role for each member node. Here
+ we assign the first node with the _member-1_ role, the second node with the
+ _member-2_ role, and the third node with the _member-3_ role:
++
roles = [
"member-1"
]
-
-.. Open the configuration/initial/module-shards.conf file and update the items listed in the following section so that the replicas match roles defined in this host’s akka.conf file.
-
++
+NOTE: This step should use a different role on each node.
++
+.. Open the configuration/initial/module-shards.conf file and update the
+ replicas so that each shard is replicated to all three nodes:
++
replicas = [
- "member-1"
+ "member-1",
+ "member-2",
+ "member-3"
]
-
-For reference, view a sample akka.conf file here: https://gist.github.com/moizr/88f4bd4ac2b03cfa45f0
-
-[start=5]
-.. Run the following commands on each of your cluster’s nodes:
-
-* *JAVA_MAX_MEM=4G JAVA_MAX_PERM_MEM=512m ./karaf*
-
-* *JAVA_MAX_MEM=4G JAVA_MAX_PERM_MEM=512m ./karaf*
-
-* *JAVA_MAX_MEM=4G JAVA_MAX_PERM_MEM=512m ./karaf*
-
-'''
-
-The OpenDaylight controller can now run in a three node cluster. Use any of the three member nodes to access the data residing in the datastore.
-
-Say you want to view information about shard designated as _member-1_ on a node. To do so, query the shard’s data by making the following HTTP request:
-
-'''
-
-*GET http://_<host>_:8181/jolokia/read/org.opendaylight.controller:Category=Shards,name=member-1-shard-inventory-config,type=DistributedConfigDatastore*
-
-NOTE: If prompted, enter _admin_ as both the username and password.
-
-'''
-
-This request should return the following information:
-
- {
- "timestamp": 1410524741,
- "status": 200,
- "request": {
- "mbean": "org.opendaylight.controller:Category=Shards,name=member-1-shard-inventory-config,type=DistributedConfigDatastore",
- "type": "read"
- },
- "value": {
- "ReadWriteTransactionCount": 0,
- "LastLogIndex": -1,
- "MaxNotificationMgrListenerQueueSize": 1000,
- "ReadOnlyTransactionCount": 0,
- "LastLogTerm": -1,
- "CommitIndex": -1,
- "CurrentTerm": 1,
- "FailedReadTransactionsCount": 0,
- "Leader": "member-1-shard-inventory-config",
- "ShardName": "member-1-shard-inventory-config",
- "DataStoreExecutorStats": {
- "activeThreadCount": 0,
- "largestQueueSize": 0,
- "currentThreadPoolSize": 1,
- "maxThreadPoolSize": 1,
- "totalTaskCount": 1,
- "largestThreadPoolSize": 1,
- "currentQueueSize": 0,
- "completedTaskCount": 1,
- "rejectedTaskCount": 0,
- "maxQueueSize": 5000
- },
- "FailedTransactionsCount": 0,
- "CommittedTransactionsCount": 0,
- "NotificationMgrExecutorStats": {
- "activeThreadCount": 0,
- "largestQueueSize": 0,
- "currentThreadPoolSize": 0,
- "maxThreadPoolSize": 20,
- "totalTaskCount": 0,
- "largestThreadPoolSize": 0,
- "currentQueueSize": 0,
- "completedTaskCount": 0,
- "rejectedTaskCount": 0,
- "maxQueueSize": 1000
- },
- "LastApplied": -1,
- "AbortTransactionsCount": 0,
- "WriteOnlyTransactionCount": 0,
- "LastCommittedTransactionTime": "1969-12-31 16:00:00.000",
- "RaftState": "Leader",
- "CurrentNotificationMgrListenerQueueStats": []
- }
- }
-
-The key thing here is the name of the shard. Shard names are structured as follows:
-
-_<member-name>_-shard-_<shard-name-as-per-configuration>_-_<store-type>_
-
-Here are a couple sample data short names:
-
-* member-1-shard-topology-config
-* member-2-shard-default-operational
-
-===== Enabling HA on a Multiple Node Cluster
-
-To enable HA in a three node cluster:
-
-'''
-
-. Open the configuration/initial/module-shards.conf file on each cluster node.
-. Add _member-2_ and _member-3_ to the replica list for each data shard.
-. Restart all of the nodes. The nodes should automatically sync up with member-1. After some time, the cluster should be ready for operation.
-
-'''
-
-When HA is enabled, you must have at least three replicas of every shard. Each node’s configuration files should look something like this:
-
- module-shards = [
- {
- name = "default"
- shards = [
- {
- name="default"
- replicas = [
- "member-1",
- "member-2",
- "member-3"
- ]
- }
- ]
- },
- {
- name = "topology"
- shards = [
- {
- name="topology"
- replicas = [
- "member-1",
- "member-2",
- "member-3"
- ]
- }
- ]
- },
- {
- name = "inventory"
- shards = [
- {
- name="inventory"
- replicas = [
- "member-1",
- "member-2",
- "member-3"
- ]
- }
- ]
- },
- {
- name = "toaster"
- shards = [
- {
- name="toaster"
- replicas = [
- "member-1",
- "member-2",
- "member-3"
- ]
- }
- ]
- }
- ]
-
-When HA is enabled on multiple nodes, shards will replicate the data for those nodes. Whenever the lead replica on a data shard is brought down, another replica takes its place. As a result, the cluster should remain available. To determine which replica is acting as the lead on a data shard, make an HTTP request to obtain the information for a data shard on any of the nodes. The resulting information will indicate which replica is acting as the lead.
-
++
+For reference, view a sample config files <<_sample_config_files,below>>.
++
+. Move into the +<karaf-distribution-directory>/bin+ directory.
+. Run the following command:
++
+ JAVA_MAX_MEM=4G JAVA_MAX_PERM_MEM=512m ./karaf
++
+. Enable clustering by running the following command at the Karaf command line:
++
+ feature:install odl-mdsal-clustering
+
+OpenDaylight should now be running in a three node cluster. You can use any of
+the three member nodes to access the data residing in the datastore.
+
+// This doesn't work at the moment. The install -s command fails.
+//===== Debugging Clustering
+//
+//To debug clustering first install Jolokia by entering the following command:
+//
+// install -s mvn:org.jolokia/jolokia-osgi/1.1.5
+//
+//After that, you can view specific information about the cluster. For example,
+//to view information about shard designated as _member-1_ on a node, query the
+//shard's data by sending the following HTTP request:
+//
+//*GET http://_<host>_:8181/jolokia/read/org.opendaylight.controller:Category=Shards,name=member-1-shard-inventory-config,type=DistributedConfigDatastore*
+//
+//NOTE: If prompted, enter your credentials for OpenDaylight. The default
+// credentials are a username and password of _admin_.
+//
+//This request should return the following information:
+//
+// {
+// "timestamp": 1410524741,
+// "status": 200,
+// "request": {
+// "mbean": "org.opendaylight.controller:Category=Shards,name=member-1-shard-inventory-config,type=DistributedConfigDatastore",
+// "type": "read"
+// },
+// "value": {
+// "ReadWriteTransactionCount": 0,
+// "LastLogIndex": -1,
+// "MaxNotificationMgrListenerQueueSize": 1000,
+// "ReadOnlyTransactionCount": 0,
+// "LastLogTerm": -1,
+// "CommitIndex": -1,
+// "CurrentTerm": 1,
+// "FailedReadTransactionsCount": 0,
+// "Leader": "member-1-shard-inventory-config",
+// "ShardName": "member-1-shard-inventory-config",
+// "DataStoreExecutorStats": {
+// "activeThreadCount": 0,
+// "largestQueueSize": 0,
+// "currentThreadPoolSize": 1,
+// "maxThreadPoolSize": 1,
+// "totalTaskCount": 1,
+// "largestThreadPoolSize": 1,
+// "currentQueueSize": 0,
+// "completedTaskCount": 1,
+// "rejectedTaskCount": 0,
+// "maxQueueSize": 5000
+// },
+// "FailedTransactionsCount": 0,
+// "CommittedTransactionsCount": 0,
+// "NotificationMgrExecutorStats": {
+// "activeThreadCount": 0,
+// "largestQueueSize": 0,
+// "currentThreadPoolSize": 0,
+// "maxThreadPoolSize": 20,
+// "totalTaskCount": 0,
+// "largestThreadPoolSize": 0,
+// "currentQueueSize": 0,
+// "completedTaskCount": 0,
+// "rejectedTaskCount": 0,
+// "maxQueueSize": 1000
+// },
+// "LastApplied": -1,
+// "AbortTransactionsCount": 0,
+// "WriteOnlyTransactionCount": 0,
+// "LastCommittedTransactionTime": "1969-12-31 16:00:00.000",
+// "RaftState": "Leader",
+// "CurrentNotificationMgrListenerQueueStats": []
+// }
+// }
+//
+//The key information is the name of the shard. Shard names are structured as follows:
+//
+//_<member-name>_-shard-_<shard-name-as-per-configuration>_-_<store-type>_
+//
+//Here are a couple sample data short names:
+//
+//* member-1-shard-topology-config
+//* member-2-shard-default-operational
+
+===== Sample Config Files
+
+.Sample +akka.conf+ file
+----
+odl-cluster-data {
+ bounded-mailbox {
+ mailbox-type = "org.opendaylight.controller.cluster.common.actor.MeteredBoundedMailbox"
+ mailbox-capacity = 1000
+ mailbox-push-timeout-time = 100ms
+ }
+
+ metric-capture-enabled = true
+
+ akka {
+ loglevel = "DEBUG"
+ loggers = ["akka.event.slf4j.Slf4jLogger"]
+
+ actor {
+
+ provider = "akka.cluster.ClusterActorRefProvider"
+ serializers {
+ java = "akka.serialization.JavaSerializer"
+ proto = "akka.remote.serialization.ProtobufSerializer"
+ }
+
+ serialization-bindings {
+ "com.google.protobuf.Message" = proto
+
+ }
+ }
+ remote {
+ log-remote-lifecycle-events = off
+ netty.tcp {
+ hostname = "10.194.189.96"
+ port = 2550
+ maximum-frame-size = 419430400
+ send-buffer-size = 52428800
+ receive-buffer-size = 52428800
+ }
+ }
+
+ cluster {
+ seed-nodes = ["akka.tcp://opendaylight-cluster-data@10.194.189.96:2550"]
+
+ auto-down-unreachable-after = 10s
+
+ roles = [
+ "member-1"
+ ]
+
+ }
+ }
+}
+
+odl-cluster-rpc {
+ bounded-mailbox {
+ mailbox-type = "org.opendaylight.controller.cluster.common.actor.MeteredBoundedMailbox"
+ mailbox-capacity = 1000
+ mailbox-push-timeout-time = 100ms
+ }
+
+ metric-capture-enabled = true
+
+ akka {
+ loglevel = "INFO"
+ loggers = ["akka.event.slf4j.Slf4jLogger"]
+
+ actor {
+ provider = "akka.cluster.ClusterActorRefProvider"
+
+ }
+ remote {
+ log-remote-lifecycle-events = off
+ netty.tcp {
+ hostname = "10.194.189.96"
+ port = 2551
+ }
+ }
+
+ cluster {
+ seed-nodes = ["akka.tcp://opendaylight-cluster-rpc@10.194.189.96:2551"]
+
+ auto-down-unreachable-after = 10s
+ }
+ }
+}
+----
+
+.Sample +module-shards.conf+ file
+----
+module-shards = [
+ {
+ name = "default"
+ shards = [
+ {
+ name="default"
+ replicas = [
+ "member-1",
+ "member-2",
+ "member-3"
+ ]
+ }
+ ]
+ },
+ {
+ name = "topology"
+ shards = [
+ {
+ name="topology"
+ replicas = [
+ "member-1",
+ "member-2",
+ "member-3"
+ ]
+ }
+ ]
+ },
+ {
+ name = "inventory"
+ shards = [
+ {
+ name="inventory"
+ replicas = [
+ "member-1",
+ "member-2",
+ "member-3"
+ ]
+ }
+ ]
+ },
+ {
+ name = "toaster"
+ shards = [
+ {
+ name="toaster"
+ replicas = [
+ "member-1",
+ "member-2",
+ "member-3"
+ ]
+ }
+ ]
+ }
+]
+----
\ No newline at end of file
Code Review / docs.git / blobdiff