1 = Cluster Wide Services
4 The existing OpenDaylight service deployment model assumes symmetric
5 clusters, where all services are activated on all nodes in the cluster.
6 However, many services require that there is a single active service
7 instance per cluster. We call such services 'singleton services'.
8 Examples of singleton services are global MD-SAL RPC services, services
9 that use centralized data processing, or the OpenFlow Topology Manager,
10 which needs to interact with all OF switches connected to a clustered
11 controller and determine how the switches are interconnected.
13 Developers of singleton services must create logic that determines
14 the active service instance, manages service failovers and ensures
15 that a service instance always runs in the surviving partition of a
16 cluster. This logic must interact with the Entity Ownership Service
17 (EOS), and it is not easy to implement and debug. Moreover, each
18 developer of an ODL-based singleton service has to design and implement
19 essentially the same functionality, each with its own behavior,
20 engineering and issues. The Cluster Singleton Service is intended to
21 abstract this funtionality into a simple-to-use service that can be
22 used by all developers of ODL-based singleton services.
25 == The General Cluster Singleton Service Approach
26 One of the key elements in the design of the Cluster Singleton Service
27 is the notion of a single cluster service instance, which corresponds to
28 a single Entity instance in the Entity Ownership Service (EOS). The EOS
29 represents the base Leadership choice for one Entity instance. Therefore,
30 candidate elections can be moved ("outsourced") to the EOS. Every Cluster
31 Singleton Service *type* must have its own Entity, and every Cluster
32 Singleton service *instance* must have its own Entity Candidate. Every
33 registered Entity Candidate must be notified about its actual role
36 To ensure that there is only one active (i.e. fully instantiated) service
37 instance in the cluster at any given time, we use the "double-candidate"
38 approach: a service instance maintains not only a candidate registration
39 for the ownership of the service's own Entity in the cluster, but also an
40 additonal (guard) ownership registration that facilitates full & graceful
41 shutdown of the service instance before the "leadership" of the service
42 is relinquished (i.e. before a service instance is deactivated). To achieve
43 "leadership" of a singleton service, a service candidate must hold ownership
44 of both these entities (see the sequence diagram below).
46 .Double Candidate Solution (Async. Close Guard)
47 include::01_doubleCandidateSimpleSequence.plantuml[]
49 The double-candidate approach prevents the shutdown of a service instance
50 with outstanding asynchronous operations, such as unfinished MD-SAL Data
51 Store transactions. The **main entity** candidate reflects the actual role
52 of the service instance in the cluster; the **close guard entity**
53 candidate is a guard that tracks outstanding asynchronous operations.
54 Every new Leader must register its own **close guard entity** candidate.
55 A Leader that wishes to relinquish its leadership must close its
56 **close guard entity** candidate. This is typically done in the last
57 step of the service's shutdown procedure. When the old Leader relinquishes
58 its **close guard entity** ownership, the new Leader will take the
59 ownership for the **close guard entity** candidate (it has to hold ownership
60 for both candidate signatures). That is the marker to full start cluster
61 node application instance and the old leader stops successfully. Figure 1
62 shows the entire sequence.
64 IMPORTANT: The double-candidate approach (with the asynchronous close
65 guard entity) creates the following prerequisite: "the actual ownership
66 doesn't change with the registration of a new candidate".
68 === The Cluster Singleton Service
69 The double-candidate solution can be used for all ODL services. Developers
70 of Singleton services no longer have to keep re-implement the same code
71 for every new service. Moreover, the Cluster Singleton Service hides the
72 interactions with the EOS service from its user. These interactions can
73 be encapsulated in an "ODL Cluster Singleton Service Provider" parent
76 .Class Diagram Cluster Singleton Service
77 include::02_classClusterSingletonService.plantuml[]
79 === Cluster Singleton Service Grouping
80 In some use cases, closely cooperating services should "share the same
81 fate", i.e. they should always be instantiated on the same Cluster Node.
82 In case of a failover, leaders for all services in the fate-sharing set
83 should be instantiated on the same (surviving) Cluster Node. For best
84 efficiency and performance, the shard leaders for shards owned by these
85 services should also reside on the same Cluster Node as the services.
86 In this case, interactions with the EOS are provided for the entire set
87 of fate-sharing ClusterSingletonService instances by a single
88 double-candidate Entity instance.
90 .Class Diagram Cluster Singleton Service Group
91 include::03_classClusterSingletonServiceGroup.plantuml[]
94 === Cluster Singleton Service Provider
95 The Provider implementation is realized as a standalone service which
96 has to be instantiated for every ClusterNode and it has to be available
97 for every dependent application. Its class diagram looks as follows.
99 .Class Diagram Cluster Singleton Service Provider
100 include::04_classClusterSingletonServiceProvider.plantuml[]
102 === Example: Cluster Singleton Service RPC Implementation
103 We'd like to show a grouping RPC service sample. RPC services don't need
104 to be a part of the same project.
108 public class SampleClusterSingletonServiceRPC_1 implements ClusterSingletonService, AutoCloseable {
110 /* Property contains an entity name guard for all instances of this group of services */
111 private static final String CLUSTER_SERVICE_GROUP_IDENTIFIER = "sample-service-group";
113 private ClusterSingletonServiceRegistration registration;
115 public SampleClusterSingletonServiceRPC_1(final ClusterSingletonServiceProvider provider) {
116 Preconditions.checkArgument(provider != null);
117 this.registration = provider.registerClusterSingletonService(this);
121 public void instantiateServiceInstance() {
122 // TODO : implement start service functionality
126 public ListenableFuture<Void> closeServiceInstance() {
127 // TODO : implement sync. or async. stop service functionality
128 return Futures.immediateFuture(null);
132 public String getServiceGroupIdentifier() {
133 return CLUSTER_SERVICE_GROUP_IDENTIFIER;
137 public void close() throws Exception {
138 if (registration != null) {
139 registration.close();
146 public class SampleClusterSingletonServiceRPC_2 implements ClusterSingletonService, AutoCloseable {
148 /* Property contains an entity name guard for all instances of this group of services */
149 private static final String CLUSTER_SERVICE_GROUP_IDENTIFIER = "sample-service-group";
151 private ClusterSingletonServiceRegistration registration;
153 public SampleClusterSingletonServiceRPC_1(final ClusterSingletonServiceProvider provider) {
154 Preconditions.checkArgument(provider != null);
155 this.registration = provider.registerClusterSingletonService(this);
159 public void instantiateServiceInstance() {
160 // TODO : implement start service functionality
164 public ListenableFuture<Void> closeServiceInstance() {
165 // TODO : implement sync. or async. stop service functionality
166 return Futures.immediateFuture(null);
170 public String getServiceGroupIdentifier() {
171 return CLUSTER_SERVICE_GROUP_IDENTIFIER;
175 public void close() throws Exception {
176 if (registration != null) {
177 registration.close();
185 Both RPCs are instantiated for some ClusterNode and both RPCs have only
186 one instance in the entire Cluster.
188 === Cluster Singleton Application
189 Applications packaged as OSGi modules can be viewed as services too. The
190 OSGI container can be viewed as an application loader. Every OSGi
191 application has its own lifecycle that should be adapted to use EOS. Only
192 the application instance that has become the "leader" should be fully
193 loaded and initialized. Basically, we would like to encapsulate
194 interactions with the EOS in a cluster-aware ODL application Loader.
196 .Life cycle of plug-ins in OSGi
197 include::05_pluginOsgiLifeCycle.plantuml[]
199 ==== Application Module Instantiation
200 Every "ODL application" contains the Provider class, which is instantiated
201 in the __AbstractModule<ODL app>__ class. A Module has the method
202 __createInstance()__, which starts an application Provider. So an application
203 provider must implement the __ClusterSingletonService__ interface and the
204 application provider initialization (or constructor) must register itself
205 to the ClusterSingletonServiceProvider. The application Provider body will
206 be initialized by the leader ClusterNode election for the master only.
208 .Base Cluster-wide app instantiation
209 include::06_baseAppSingleInstance.plantuml[]
211 So we are able to hide whole EOS interaction from the user and encapsulate
212 it inside the "ClusterSingletonServiceProvider" implementation. An application
213 or a service only needs to implement the relevant interface and register
214 itself to the ??? provider.
216 A simplified sequence diagram (without the double-candidate) is displayed
217 in the following figure:
219 .Simply Cluster-wide app instantiation (without double candidate)
220 include::07_processAppSingleInstSimply.plantuml[]
222 The full sequence implementation diagram for the __AbstractClusterProjectProvider__ ??? is displayed in the following figure:
224 .Cluster-wide app instantiation
225 include::08_processAppSingleInst.plantuml[]
229 public class ClusterSingletonProjectSample implements ClusterSingletonService, AutoCloseable {
231 /* Property contains an entity name guard for all instances of this group of services */
232 private static final String CLUSTER_SERVICE_GROUP_IDENTIFIER = "sample-service-group";
234 private ClusterSingletonServiceRegistration registration;
236 public ClusterSingletonProjectSample(final ClusterSingletonServiceProvider provider) {
237 Preconditions.checkArgument(provider != null);
238 this.registration = provider.registerClusterSingletonService(this);
242 public void instantiateServiceInstance() {
243 // TODO : implement start project functionality
248 public ListenableFuture<Void> closeServiceInstance() {
249 // TODO : implement sync. or async. stop project functionality
250 return Futures.immediateFuture(null);
254 public String getServiceGroupIdentifier() {
255 return CLUSTER_SERVICE_GROUP_IDENTIFIER;
259 public void close() throws Exception {
260 if (registration != null) {
261 registration.close();
268 public class ApplicationModule extends ProjectAbstractModule<? extends AbstractStatisticsManagerModule> {
273 public java.lang.AutoCloseable createInstance() {
274 AbstractServiceProvider projectProvider =
275 new ClusterSingletonProjectSample(getClusterSingletonServiceProviderDependency());
276 return projectProvider;