+= Cluster Wide Services
+
+----
+The existing OpenDaylight service deployment model assumes symmetric
+clusters, where all services are activated on all nodes in the cluster.
+However, many services require that there is a single active service
+instance per cluster. Examples are global MD-SAL RPC services, or
+services that use centralized data processing, or the OpenFlow Topology
+Manager, which needs to interact with all OF switches connected to a
+clustered controller and determine how the switches are interconnected.
+We call such services 'singleton services'.
+
+A developer of a singleton service must create logic that determines
+the active service instance, manages service failovers and ensures that
+a service instance always runs in the surviving partition of a cluster.
+This logic would have to interact with the Entity Ownership Service (EOS)
+and it is not easy to get it right. Leaving it to individual services
+would mean that each service would design and implement essentially the
+same functionality, each with its own behavior, engineering and issues.
+----
+
+== General Cluster Singleton Service Approach
+The main idea represents a single cluster service instance. The Entity
+Ownership Service (EOS) represents the base Leadership choice for one
+Entity instance. So we are able to move candidate election to the EOS.
+Every Cluster Singleton service *type* must have its own Entity and every
+Cluster Singleton service *instance* must have its own Entity Candidate.
+Every registered Entity Candidate should be notified about its actual role
+in the cluster.
+
+To ensure that there is only one active (i.e. fully instantiated) service
+instance in the cluster at any given time, we use the "double-candidate"
+approach: a service instance maintains not only a candidate registration
+for the ownership of the service's Entity in the cluster, but also an
+additonal (guard) ownership registration that ensures a full shutdown of
+the service instance before the overall ownership of the service is
+relinquished. To achieve the overall ownership of a singleton service,
+a service candidate must hold ownership of both these entities (see the
+sequence diagram below).
+
+.Double Candidate Solution (Async. Close Guard)
+include::01_doubleCandidateSimpleSequence.plantuml[]
+
+The double-candidate approach prevents the shutdown of a service with
+outstanding asynchronous operations, such as unfinished MD-SAL Data
+Store transactions. The **main entity** candidate is focused on the
+actual role of the service in the cluster; the **close guard entity**
+candidate is a guard that tracks the outstanding asynchronous operations.
+Every new Leader must register its own **close guard entity** candidate.
+A Leader that wishes to relinquish its leadership must close its
+**close guard entity** candidate. This is typically done in the last
+step of the shutdown procedure. When the old Leader relinquishes its
+**close guard entity** ownership, the new Leader will take the leadership
+for the **close guard entity** candidate (it has to hold ownership for
+both candidate signatures). That is the marker to full start cluster
+node application instance and old leader stops successfully. Figure 1
+shows the entire sequence.
+
+IMPORTANT: Double candidate approach (async. close guard) prerequisite is "actual ownership doesn't change by new candidate registration".
+
+=== Cluster Singleton Service
+Double candidate solution is relevant for all services and we don't need to implement same code for every instances. So we are able to hide whole EOS interaction for user and we can encapsulate it to some "ODL Cluster Singleton Service Provider" parent.
+
+.Class Diagram Cluster Singleton Service
+include::02_classClusterSingletonService.plantuml[]
+
+=== Cluster Singleton Service Grouping
+Sometimes we wish to have couple of services to run on same Cluster Node. So Double candidate EOS interaction could by realized for a list of ClusterSingletonService instances.
+
+.Class Diagram Cluster Singleton Service Group
+include::03_classClusterSingletonServiceGroup.plantuml[]
+
+
+=== Cluster Singleton Service Provider
+Provider implementation is realized as stay alone service which has to be instantiated for every ClusterNode and it has to be available for every depend applications. So class diagram looks as next.
+
+.Class Diagram Cluster Singleton Service Provider
+include::04_classClusterSingletonServiceProvider.plantuml[]
+
+=== Cluster Singleton Service RPC implementation sample
+We'd like to show a grouping RPC service sample. RPC services don't need be a part of same project.
+
+[source,java]
+----
+public class SampleClusterSingletonServiceRPC_1 implements ClusterSingletonService, AutoCloseable {
+
+ /* Property contains an entity name guard for all instances of this group of services */
+ private static final String CLUSTER_SERVICE_GROUP_IDENTIFIER = "sample-service-group";
+
+ private ClusterSingletonServiceRegistration registration;
+
+ public SampleClusterSingletonServiceRPC_1(final ClusterSingletonServiceProvider provider) {
+ Preconditions.checkArgument(provider != null);
+ this.registration = provider.registerClusterSingletonService(this);
+ }
+
+ @Override
+ public void instantiateServiceInstance() {
+ // TODO : implement start service functionality
+ }
+
+ @Override
+ public ListenableFuture<Void> closeServiceInstance() {
+ // TODO : implement sync. or async. stop service functionality
+ return Futures.immediateFuture(null);
+ }
+
+ @Override
+ public String getServiceGroupIdentifier() {
+ return CLUSTER_SERVICE_GROUP_IDENTIFIER;
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (registration != null) {
+ registration.close();
+ registration = null;
+ }
+ }
+
+}
+
+public class SampleClusterSingletonServiceRPC_2 implements ClusterSingletonService, AutoCloseable {
+
+ /* Property contains an entity name guard for all instances of this group of services */
+ private static final String CLUSTER_SERVICE_GROUP_IDENTIFIER = "sample-service-group";
+
+ private ClusterSingletonServiceRegistration registration;
+
+ public SampleClusterSingletonServiceRPC_1(final ClusterSingletonServiceProvider provider) {
+ Preconditions.checkArgument(provider != null);
+ this.registration = provider.registerClusterSingletonService(this);
+ }
+
+ @Override
+ public void instantiateServiceInstance() {
+ // TODO : implement start service functionality
+ }
+
+ @Override
+ public ListenableFuture<Void> closeServiceInstance() {
+ // TODO : implement sync. or async. stop service functionality
+ return Futures.immediateFuture(null);
+ }
+
+ @Override
+ public String getServiceGroupIdentifier() {
+ return CLUSTER_SERVICE_GROUP_IDENTIFIER;
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (registration != null) {
+ registration.close();
+ registration = null;
+ }
+ }
+
+}
+----
+
+Both RPCs are instantiated for some ClusterNode and RPCs have only one instance in whole Cluster.
+
+=== Cluster Singleton Application
+OSGi module application could be understand like service too. So we would like to focus for OSGi container like a application loader. Every OSGi app has own lifecycle which should be adapting to use EOS and only master could be loading fully. We wish to encapsulate EOS interaction in an ODL application Loader.
+
+.Life cycle of plug-ins in OSGi
+include::05_pluginOsgiLifeCycle.plantuml[]
+
+==== Application Module instantiation
+Every "ODL app." has Provider class which is instantiated in __AbstractModule<ODL app>__ class. Module has method __createInstance()__ which start an application Provider. So application provider has to implement __ClusterSingletonService__ interface and the application provider initialization (or constructor) has to register itself to ClusterSingletonServiceProvider. The application Provider body will be initialized by leader ClusterNode election for master only.
+
+.Base Cluster-wide app instantiation
+include::06_baseAppSingleInstance.plantuml[]
+
+So we are able to hide whole EOS interaction for user and encapsulate inside "ClusterSingletonServiceProvider". Application/service needs only implement relevant interface and registrates itself to provider.
+
+Simplified sequence diagram (without double candidate) is displayed in next picture:
+
+.Simply Cluster-wide app instantiation (without double candidate)
+include::07_processAppSingleInstSimply.plantuml[]
+
+Full sequence implementation diagram for __AbstractClusterProjectProvider__ is displayed in next picture:
+
+.Cluster-wide app instantiation
+include::08_processAppSingleInst.plantuml[]
+
+[source,java]
+----
+public class ClusterSingletonProjectSample implements ClusterSingletonService, AutoCloseable {
+
+ /* Property contains an entity name guard for all instances of this group of services */
+ private static final String CLUSTER_SERVICE_GROUP_IDENTIFIER = "sample-service-group";
+
+ private ClusterSingletonServiceRegistration registration;
+
+ public ClusterSingletonProjectSample(final ClusterSingletonServiceProvider provider) {
+ Preconditions.checkArgument(provider != null);
+ this.registration = provider.registerClusterSingletonService(this);
+ }
+
+ @Override
+ public void instantiateServiceInstance() {
+ // TODO : implement start project functionality
+
+ }
+
+ @Override
+ public ListenableFuture<Void> closeServiceInstance() {
+ // TODO : implement sync. or async. stop project functionality
+ return Futures.immediateFuture(null);
+ }
+
+ @Override
+ public String getServiceGroupIdentifier() {
+ return CLUSTER_SERVICE_GROUP_IDENTIFIER;
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (registration != null) {
+ registration.close();
+ registration = null;
+ }
+ }
+
+}
+
+public class ApplicationModule extends ProjectAbstractModule<? extends AbstractStatisticsManagerModule> {
+
+ ...
+
+ @Override
+ public java.lang.AutoCloseable createInstance() {
+ AbstractServiceProvider projectProvider =
+ new ClusterSingletonProjectSample(getClusterSingletonServiceProviderDependency());
+ return projectProvider;
+ }
+}
+----