1 == Setting Up Clustering
3 === Clustering Overview
5 Clustering is a mechanism that enables multiple processes and programs to work
6 together as one entity. For example, when you search for something on
7 google.com, it may seem like your search request is processed by only one web
8 server. In reality, your search request is processed by may web servers
9 connected in a cluster. Similarly, you can have multiple instances of
10 OpenDaylight working together as one entity.
12 Advantages of clustering are:
14 * Scaling: If you have multiple instances of OpenDaylight running, you can
15 potentially do more work and store more data than you could with only one
16 instance. You can also break up your data into smaller chunks (shards) and
17 either distribute that data across the cluster or perform certain operations
18 on certain members of the cluster.
19 * High Availability: If you have multiple instances of OpenDaylight running and
20 one of them crashes, you will still have the other instances working and
22 * Data Persistence: You will not lose any data stored in OpenDaylight after a
23 manual restart or a crash.
25 The following sections describe how to set up clustering on both individual and
26 multiple OpenDaylight instances.
28 === Single Node Clustering
30 To enable clustering on a single instance of OpenDaylight, perform the
33 . Download, unzip, and run the OpenDaylight distribution
34 . Install the clustering feature:
36 feature:install odl-mdsal-clustering
38 NOTE: This will enabled the cluster-ready version of the MD-SAL data store, but
39 will not actually create a cluster of multiple instances. The result is
40 that you will get data persistence, but not the scaling or high
41 availability advantages.
43 === Multiple Node Clustering
45 The following sections describe how to set up multiple node clusters in OpenDaylight.
47 ==== Deployment Considerations
49 To implement clustering, the deployment considerations are as follows:
51 * To set up a cluster with multiple nodes, we recommend that you use a minimum
52 of three machines. You can set up a cluster with just two nodes. However, if
53 one of the two nodes fail, the cluster will not be operational.
55 NOTE: This is because clustering in OpenDaylight requires a majority of the
56 nodes to be up and one node cannot be a majority of two nodes.
58 * Every device that belongs to a cluster needs to have an identifier.
59 OpenDaylight uses the node's +role+ for this purpose. After you define the
60 first node's role as _member-1_ in the +akka.conf+ file, OpenDaylight uses
61 _member-1_ to identify that node.
63 * Data shards are used to contain all or a certain segment of a OpenDaylight's
64 MD-SAL datastore. For example, one shard can contain all the inventory data
65 while another shard contains all of the topology data.
67 If you do not specify a module in the +modules.conf+ file and do not specify
68 a shard in +module-shards.conf+, then (by default) all the data is placed in
69 the default shard (which must also be defined in +module-shards.conf+ file).
70 Each shard has replicas configured. You can specify the details of where the
71 replicas reside in +module-shards.conf+ file.
73 * If you have a three node cluster and would like to be able to tolerate any
74 single node crashing, a replica of every defined data shard must be running
75 on all three cluster nodes.
77 NOTE: This is because OpenDaylight's clustering implementation requires a
78 majority of the defined shard replicas to be running in order to
79 function. If you define data shard replicas on two of the cluster nodes
80 and one of those nodes goes down, the corresponding data shards will not
83 * If you have a three node cluster and have defined replicas for a data shard
84 on each of those nodes, that shard will still function even if only two of
85 the cluster nodes are running. Note that if one of those remaining two nodes
86 goes down, the shard will not be operational.
88 * It is recommended that you have multiple seed nodes configured. After a
89 cluster member is started, it sends a message to all of its seed nodes.
90 The cluster member then sends a join command to the first seed node that
91 responds. If none of its seed nodes reply, the cluster member repeats this
92 process until it successfully establishes a connection or it is shut down.
94 * After a node is unreachable, it remains down for configurable period of time
95 (10 seconds, by default). Once a node goes down, you need to restart it so
96 that it can rejoin the cluster. Once a restarted node joins a cluster, it
97 will synchronize with the lead node automatically.
99 ==== Setting Up a Multiple Node Cluster
101 To run OpenDaylight in a three node cluster, perform the following:
103 First, determine the three machines that will make up the cluster. After that,
104 do the following on each machine:
106 . Copy the OpenDaylight distribution zip file to the machine.
107 . Unzip the distribution.
108 . Open the following .conf files:
109 ** configuration/initial/akka.conf
110 ** configuration/initial/module-shards.conf
111 . In each configuration file, make the following changes:
112 .. Find every instance of the following lines and replace _127.0.0.1_ with the
113 hostname or IP address of the machine on which this file resides and
114 OpenDaylight will run:
117 hostname = "127.0.0.1"
119 NOTE: The value you need to specify will be different for each node in the
122 .. Find the following lines and replace _127.0.0.1_ with the hostname or IP
123 address of any of the machines that will be part of the cluster:
126 seed-nodes = ["akka.tcp://opendaylight-cluster-data@127.0.0.1:2550"]
128 .. Find the following section and specify the role for each member node. Here
129 we assign the first node with the _member-1_ role, the second node with the
130 _member-2_ role, and the third node with the _member-3_ role:
136 NOTE: This step should use a different role on each node.
138 .. Open the configuration/initial/module-shards.conf file and update the
139 replicas so that each shard is replicated to all three nodes:
147 For reference, view a sample config files <<_sample_config_files,below>>.
149 . Move into the +<karaf-distribution-directory>/bin+ directory.
150 . Run the following command:
152 JAVA_MAX_MEM=4G JAVA_MAX_PERM_MEM=512m ./karaf
154 . Enable clustering by running the following command at the Karaf command line:
156 feature:install odl-mdsal-clustering
158 OpenDaylight should now be running in a three node cluster. You can use any of
159 the three member nodes to access the data residing in the datastore.
161 // This doesn't work at the moment. The install -s command fails.
162 //===== Debugging Clustering
164 //To debug clustering first install Jolokia by entering the following command:
166 // install -s mvn:org.jolokia/jolokia-osgi/1.1.5
168 //After that, you can view specific information about the cluster. For example,
169 //to view information about shard designated as _member-1_ on a node, query the
170 //shard's data by sending the following HTTP request:
172 //*GET http://_<host>_:8181/jolokia/read/org.opendaylight.controller:Category=Shards,name=member-1-shard-inventory-config,type=DistributedConfigDatastore*
174 //NOTE: If prompted, enter your credentials for OpenDaylight. The default
175 // credentials are a username and password of _admin_.
177 //This request should return the following information:
180 // "timestamp": 1410524741,
183 // "mbean": "org.opendaylight.controller:Category=Shards,name=member-1-shard-inventory-config,type=DistributedConfigDatastore",
187 // "ReadWriteTransactionCount": 0,
188 // "LastLogIndex": -1,
189 // "MaxNotificationMgrListenerQueueSize": 1000,
190 // "ReadOnlyTransactionCount": 0,
191 // "LastLogTerm": -1,
192 // "CommitIndex": -1,
194 // "FailedReadTransactionsCount": 0,
195 // "Leader": "member-1-shard-inventory-config",
196 // "ShardName": "member-1-shard-inventory-config",
197 // "DataStoreExecutorStats": {
198 // "activeThreadCount": 0,
199 // "largestQueueSize": 0,
200 // "currentThreadPoolSize": 1,
201 // "maxThreadPoolSize": 1,
202 // "totalTaskCount": 1,
203 // "largestThreadPoolSize": 1,
204 // "currentQueueSize": 0,
205 // "completedTaskCount": 1,
206 // "rejectedTaskCount": 0,
207 // "maxQueueSize": 5000
209 // "FailedTransactionsCount": 0,
210 // "CommittedTransactionsCount": 0,
211 // "NotificationMgrExecutorStats": {
212 // "activeThreadCount": 0,
213 // "largestQueueSize": 0,
214 // "currentThreadPoolSize": 0,
215 // "maxThreadPoolSize": 20,
216 // "totalTaskCount": 0,
217 // "largestThreadPoolSize": 0,
218 // "currentQueueSize": 0,
219 // "completedTaskCount": 0,
220 // "rejectedTaskCount": 0,
221 // "maxQueueSize": 1000
223 // "LastApplied": -1,
224 // "AbortTransactionsCount": 0,
225 // "WriteOnlyTransactionCount": 0,
226 // "LastCommittedTransactionTime": "1969-12-31 16:00:00.000",
227 // "RaftState": "Leader",
228 // "CurrentNotificationMgrListenerQueueStats": []
232 //The key information is the name of the shard. Shard names are structured as follows:
234 //_<member-name>_-shard-_<shard-name-as-per-configuration>_-_<store-type>_
236 //Here are a couple sample data short names:
238 //* member-1-shard-topology-config
239 //* member-2-shard-default-operational
241 ===== Sample Config Files
243 .Sample +akka.conf+ file
247 mailbox-type = "org.opendaylight.controller.cluster.common.actor.MeteredBoundedMailbox"
248 mailbox-capacity = 1000
249 mailbox-push-timeout-time = 100ms
252 metric-capture-enabled = true
256 loggers = ["akka.event.slf4j.Slf4jLogger"]
260 provider = "akka.cluster.ClusterActorRefProvider"
262 java = "akka.serialization.JavaSerializer"
263 proto = "akka.remote.serialization.ProtobufSerializer"
266 serialization-bindings {
267 "com.google.protobuf.Message" = proto
272 log-remote-lifecycle-events = off
274 hostname = "10.194.189.96"
276 maximum-frame-size = 419430400
277 send-buffer-size = 52428800
278 receive-buffer-size = 52428800
283 seed-nodes = ["akka.tcp://opendaylight-cluster-data@10.194.189.96:2550"]
285 auto-down-unreachable-after = 10s
297 mailbox-type = "org.opendaylight.controller.cluster.common.actor.MeteredBoundedMailbox"
298 mailbox-capacity = 1000
299 mailbox-push-timeout-time = 100ms
302 metric-capture-enabled = true
306 loggers = ["akka.event.slf4j.Slf4jLogger"]
309 provider = "akka.cluster.ClusterActorRefProvider"
313 log-remote-lifecycle-events = off
315 hostname = "10.194.189.96"
321 seed-nodes = ["akka.tcp://opendaylight-cluster-rpc@10.194.189.96:2551"]
323 auto-down-unreachable-after = 10s
329 .Sample +module-shards.conf+ file