Merge "Fixing PacketCable user guide formatting" into stable/lithium

[docs.git] / manuals / user-guide / src / main / asciidoc / ch-clustering.adoc

== Setting Up Clustering on an OpenDaylight Controller

//* <<Clustering Overview>>
//* <<Single Node Clustering>>
//* <<Multiple Node Clustering>>
//** <<Deployment Considerations>>
//** <<Setting Up a Multiple Node Cluster>>
//*** <<Enabling HA on a Multiple Node Cluster>>

=== Clustering Overview

Clustering is a mechanism that enables multiple processes and programs to work together as one entity.  For example, when you go to google.com and search for something, it may seem like your search request is processed by only one web server. In reality, your search request is processed by thousands of web servers connected in a cluster. Similarly, you can have multiple instances of the OpenDaylight controller working together as one entity. There are a number of uses for clustering:

* Scaling: If you have multiple controllers running, you can potentially do more work with or store more data on those controllers if they are clustered. You can also break up your data into smaller chunks (known as shards) and either distribute that data across the cluster or perform certain operations on certain members of the cluster.

* High Availability: If you have multiple controllers running and one of them crashes, you would still have the other instances working and available.

* Data Persistence: You will not lose any data gathered by your controller after a manual restart or a crash.

The following sections describe how to set up clustering on both individual and multiple OpenDaylight controllers.

=== Single Node Clustering
To enable clustering on a single OpenDaylight controller, do the following:

'''

. Download and unzip a base controller distribution. You must use the new OpenFlow plugin, so download a distribution where the new OpenFlow plugin is either the default or can be enabled.
. Navigate to the _<Karaf-distribution-location>_/bin directory.
. Run Karaf: *./karaf*
. Install the clustering feature: *feature:install odl-mdsal-clustering*
. If you are using the integration distribution of Karaf, you should also install the open flow plugin flow services: *feature:install odl-openflowplugin-flow-services*
. Install the Jolokia bundle: *install -s mvn:org.jolokia/jolokia-osgi/1.1.5*

'''

After enabling the DistributedDataStore feature in a single instance, you can access the following features:

* Data Sharding: The in-memory MD-SAL tree is broken up into a number of smaller sub-trees (inventory, topology, and default).
* Data Persistence: All of the data available on defined data shards is stored on a disk. By restarting the controller, you can use the persisted data to reinstate those shards to their previous state.

=== Multiple Node Clustering

The following sections describe how to set up multiple node clusters in OpenDaylight.

==== Deployment Considerations

Here is some information to keep in mind when you implement clustering:

* When setting up a cluster with multiple nodes, we recommend that you do so with a minimum of three machines. You can set up with a cluster with just two nodes. However, if one of those two nodes go down, the controller will no longer be operational.

* Every device that belongs to a cluster needs to have an identifier. For this purpose, OpenDaylight uses the node&#8217;s role. After you define the first node&#8217;s role as _member-1_ in the akka.conf file, OpenDaylight uses _member-1_ to identify that node.

* Data shards are used to house all or a certain segment of a module&#8217;s data. For example, one shard can contain all of a module&#8217;s inventory data while another shard contains all of it&#8217;s topology data. If you do not specify a module in the modules.conf file and do not specify a shard in module-shards.conf, then (by default) all the data is places onto the default shard (which must also be defined in module-shards.conf file). Each shard has replicas configured, and the module-shards.conf file is where you can specify where these replicas reside.

* Say you have a three node cluster on which HA is enabled. A replica of every defined data shard must be running on all three cluster nodes. This is because OpenDaylight&#8217;s clustering implementation requires a majority of the defined shard replicas to be running in order to function. If you only define data shard replicas on two of the cluster nodes and one of those nodes goes down, the corresponding data shards will not function.

* If you have a three node cluster and have defined replicas for a data shard on each of those nodes, that shard will still function even if only two of the cluster nodes are running. Note that if one of those two nodes go down, your controller will no longer be operational.

*What considerations need to be made when setting the seed nodes for each member? Why are we referring to multiple seed nodes when you set only one IP address? Can you set multiple seed nodes for functional testing?*

We recommend that you have multiple seed nodes configured. After a cluster member is started, it sends a message to all of its seed nodes. The cluster member then sends a join command to the first seed node that responds. If none of its seed nodes reply, the cluster member repeats this process until it successfully establishes a connection or it is shutdown.

*What happens after one node becomes unreachable? Do the other two nodes function normally? When the first node reconnects, does it automatically synchronize with the other nodes?*

After a node becomes unreachable, it remains down for configurable period of time (10 seconds, by default). Once a node goes down, you need to restart it so that it can rejoin the cluster. Once a restarted node joins a cluster, it will synchronize with the lead node automatically.

*Can you run a two node cluster for functional testing?*

For functional testing, yes. For HA testing, you need to run all three nodes.

==== Setting Up a Multiple Node Cluster

To run an OpenDaylight controller in a three node cluster, do the following:

'''

. Determine the three machines that will make up the cluster and copy the controller distribution to each of those machines.
. Unzip the controller distribution.
. Navigate to the _<Karaf-distribution-location>_/bin directory.
. Run Karaf: *./karaf*
. Install the clustering feature: *feature:install odl-mdsal-clustering*

NOTE: To run clustering, you must install the odl-mdsal-clustering feature on each of your nodes.

[start=6]
. If you are using the integration distribution of Karaf, you should also install the open flow plugin flow services: *feature:install odl-openflowplugin-flow-services*
. Install the Jolokia bundle: *install -s mvn:org.jolokia/jolokia-osgi/1.1.5*
. On each node, open the following .conf files:

* configuration/initial/akka.conf
* configuration/initial/module-shards.conf

. In each configuration file, make the following changes:

.. Find every instance of the following lines and replace _127.0.0.1_ with the hostname or IP address of the machine on which the controller will run:

     netty.tcp {
       hostname = "127.0.0.1"

NOTE: The value you need to specify will be different for each node in the cluster.

[start=2]
.. Find the following lines and replace _127.0.0.1_ with the hostname or IP address of any of the machines that will be part of the cluster:

   cluster {
     seed-nodes = ["akka.tcp://opendaylight-cluster-data@127.0.0.1:2550"]

.. Find the following section and specify the role for each member node. For example, you could assign the first node with the _member-1_ role, the second node with the _member-2_ role, and the third node with the _member-3_ role.


     roles = [
       "member-1"
     ]

.. Open the configuration/initial/module-shards.conf file and update the items listed in the following section so that the replicas match roles defined in this host&#8217;s akka.conf file.

               replicas = [
                   "member-1"
               ]

For reference, view a sample akka.conf file here: https://gist.github.com/moizr/88f4bd4ac2b03cfa45f0

[start=5]
.. Run the following commands on each of your cluster&#8217;s nodes:

* *JAVA_MAX_MEM=4G JAVA_MAX_PERM_MEM=512m ./karaf*

* *JAVA_MAX_MEM=4G JAVA_MAX_PERM_MEM=512m ./karaf*

* *JAVA_MAX_MEM=4G JAVA_MAX_PERM_MEM=512m ./karaf*

'''

The OpenDaylight controller can now run in a three node cluster. Use any of the three member nodes to access the data residing in the datastore.

Say you want to view information about shard designated as _member-1_ on a node. To do so, query the shard&#8217;s data by making the following HTTP request:

'''

*GET http://_<host>_:8181/jolokia/read/org.opendaylight.controller:Category=Shards,name=member-1-shard-inventory-config,type=DistributedConfigDatastore*

NOTE: If prompted, enter _admin_ as both the username and password.

'''

This request should return the following information:

   {
       "timestamp": 1410524741,
       "status": 200,
       "request": {
       "mbean": "org.opendaylight.controller:Category=Shards,name=member-1-shard-inventory-config,type=DistributedConfigDatastore",
       "type": "read"
       },
       "value": {
       "ReadWriteTransactionCount": 0,
       "LastLogIndex": -1,
       "MaxNotificationMgrListenerQueueSize": 1000,
       "ReadOnlyTransactionCount": 0,
       "LastLogTerm": -1,
       "CommitIndex": -1,
       "CurrentTerm": 1,
       "FailedReadTransactionsCount": 0,
       "Leader": "member-1-shard-inventory-config",
       "ShardName": "member-1-shard-inventory-config",
       "DataStoreExecutorStats": {
       "activeThreadCount": 0,
       "largestQueueSize": 0,
       "currentThreadPoolSize": 1,
       "maxThreadPoolSize": 1,
       "totalTaskCount": 1,
       "largestThreadPoolSize": 1,
       "currentQueueSize": 0,
       "completedTaskCount": 1,
       "rejectedTaskCount": 0,
       "maxQueueSize": 5000
       },
       "FailedTransactionsCount": 0,
       "CommittedTransactionsCount": 0,
       "NotificationMgrExecutorStats": {
       "activeThreadCount": 0,
       "largestQueueSize": 0,
       "currentThreadPoolSize": 0,
       "maxThreadPoolSize": 20,
       "totalTaskCount": 0,
       "largestThreadPoolSize": 0,
       "currentQueueSize": 0,
       "completedTaskCount": 0,
       "rejectedTaskCount": 0,
       "maxQueueSize": 1000
       },
       "LastApplied": -1,
       "AbortTransactionsCount": 0,
       "WriteOnlyTransactionCount": 0,
       "LastCommittedTransactionTime": "1969-12-31 16:00:00.000",
       "RaftState": "Leader",
       "CurrentNotificationMgrListenerQueueStats": []
       }
   }

The key thing here is the name of the shard. Shard names are structured as follows:

_<member-name>_-shard-_<shard-name-as-per-configuration>_-_<store-type>_

Here are a couple sample data short names:

* member-1-shard-topology-config
* member-2-shard-default-operational

===== Enabling HA on a Multiple Node Cluster

To enable HA in a three node cluster:

'''

. Open the configuration/initial/module-shards.conf file on each cluster node.
. Add _member-2_ and _member-3_ to the replica list for each data shard.
. Restart all of the nodes. The nodes should automatically sync up with member-1. After some time, the cluster should be ready for operation.

'''

When HA is enabled, you must have at least three replicas of every shard. Each node&#8217;s configuration files should look something like this:

   module-shards = [
       {
           name = "default"
           shards = [
               {
                   name="default"
                   replicas = [
                       "member-1",
                       "member-2",
                       "member-3"
                   ]
               }
           ]
       },
       {
           name = "topology"
           shards = [
               {
                   name="topology"
                   replicas = [
                       "member-1",
                       "member-2",
                       "member-3"
                   ]
               }
           ]
       },
       {
           name = "inventory"
           shards = [
               {
                   name="inventory"
                   replicas = [
                       "member-1",
                       "member-2",
                       "member-3"
                   ]
               }
           ]
       },
       {
            name = "toaster"
            shards = [
                {
                    name="toaster"
                    replicas = [
                       "member-1",
                       "member-2",
                       "member-3"
                    ]
                }
            ]
       }
   ]

When HA is enabled on multiple nodes, shards will replicate the data for those nodes. Whenever the lead replica on a data shard is brought down, another replica takes its place. As a result, the cluster should remain available. To determine which replica is acting as the lead on a data shard, make an HTTP request to obtain the information for a data shard on any of the nodes. The resulting information will indicate which replica is acting as the lead.