Merge "Splitted controller.adoc into separate files"

[docs.git] / manuals / developer-guide / src / main / asciidoc / controller / md-sal.adoc

=== OpenDaylight Controller: MD-SAL Developers' Guide

Model-Driven SAL (MD-SAL) is a set of infrastructure services aimed at providing common and generic support to application and plugin developers.

MD-SAL currently provides infrastructure services for the following:

* Data Services
* RPC or Service routing
* Notification subscription and publish services

This model-driven infrastructure allows developers to develop applications and plugins against an API type of their choice (Java generated APIs, DOM APIs, REST APIs). The infrastructure automatically provides the other API types.
The modelling language of choice for MD-SAL is YANG, which is an IETF standard, for modelling network element configuration. The YANGTools project and its development tools provide support for YANG.


=== API types

MD-SAL provides three API types: +

* Java generated APIs for consumers and producers
* DOM APIs: Mostly used by infrastucture components and usuful for XML-driven plugin and application types
* REST APIs: https://wiki.opendaylight.org/view/OpenDaylight_Controller:MD-SAL:Restconf[Restconf] that is available to consumer type applications and provides access to RPC and data stores


=== Basic YANG concepts and their rendition in APIs

The following are the basic concepts in YANG modeling: +

* Remote Procedure (RPCs): In MD-SAL, RPCs are used for any call or invocation that crosses the plugin or module boundaries. RPCs are triggered by consumers, and usually have return values.
* Notifications: Asynchronous events, published by components for listeners.
* Configuration and Operational Data tree: The well-defined (by model) tree structure that represents the operational state of components and systems.
** Instance Identifier: The path that uniquely identifies the sub-tree in the configuration or operational space. Most of the addressing of data is done by Instance Identifier.

==== RPC
In YANG, Remote Procedure Calls (RPCs) are used to model any procedure call implemented by a Provider (Server), which exposes functionality to Consumers (Clients).

In MD-SAL terminology, the term 'RPC' is used to define the input and output for a procedure (function) that is to be provided by a Provider, and adapted by the MD-SAL.

In the context of the MD-SAL, there are three types of RPCs (RPC services): +

* Global: One service instance (implementation) per controller container or mount point
* Routed: Multiple service instances (implementations) per controller container or mount point

==== Global service

* There is only one instance of a Global Service per controller instance. (Note that a controller instance can consist of a cluster of controller nodes.)

*Routing* +

* Binding-Aware MD-SAL (sal-binding)
** **Rpc Type**: Identified by a generated RpcService class and a name of a method invoked on that interface
* Binding-Independent MD-SAL (sal-dom)
** **Rpc Type**: Identified by a QName

==== Routed service ====

* There can be multiple instances (implementations) of a service per controller instance
* Can be used for southbound plugins or for horizontal scaling (load-balancing) of northbound plugins (services)

*Routing* +

Routing is done based on the contents of a message, for example, 'Node Reference'. The field in a message that is used for routing is specified in a YANG model by using the routing-reference statement from the yang-ext model. +

* Binding Aware MD-SAL (sal-binding)
* RPC Type: Identified by an RpcService subclass and the name of the method invoked on that interface
* Instance Identifier: In a data tree, identifies the element instance that will be used as the route target.
The used class is: +
----
org.opendaylight.yang.binding.InstanceIdentifier
----

The Instance Identifier is learned from the message payload and from the model. +

* Binding Independent MD-SAL (sal-dom)
* RPC Type: Identified by a QName

* Instance Identifier: In a data tree, identifies the element instance that will be used as the route target. The used class is: +
----
org.opendaylight.yang.data.api.InstanceIdentifier
----
RPCs in various API types: +

* Java Generated APIs: For each model there is *Service interface. See https://wiki.opendaylight.org/view/YANG_Tools:YANG_to_Java_Mapping#Rpc[YANG Tools: Yang to Java mapping-RPC]  to understand how YANG statements maps to Service interface.
** Providers expose their implementation of *Service by registering their implementation to RpcProviderRegistry.
** Consumers get the *Service implementation from RpcConsumerRegistry. If the implementer uses a different API type, MD-SAL automatically translates data in the background.
* DOM APIs: RPCs are identified by QName.
** Providers expose their implementation of RPC identified by QName registering their RpcImplementation to RpcProvisionRegistry.
** Consumers get the *Service implementation from RpcConsumerRegistry. If the implementer uses different API type, MD-SAL automatically translates data in the background.
* REST APIs: RPCs are identified by the model name and their name.
* Consumers invoke RPCs by invoking POST operation to /restconf/operations/model-name:rpc-name.

==== Notification
In YANG, Notifications represent asynchronous events, published by providers for listeners.

RPCs in various API types: +

* Java Generated APIs: For each model, there is *Listener interface and transfer object for each notification. See https://wiki.opendaylight.org/view/YANG_Tools:YANG_to_Java_Mapping#Notification[YANG Tools: Yang to Java mapping-Notification] to understand how YANG statements map to the Notifications interface.
** Providers publish notifications by invoking the publish method on NotificationPublishService.
** To receive notifications, consumers register their implementation of *Listener to NotificationBrokerService. If the notification publisher uses a different API type, MD-SAL automatically translates data in the background.
* DOM APIs: Notifications are represented only by XML Payload.
** Providers publish notifications by invoking the publish method on NotificationPublishService.
** To receive notifications, consumers register their implementation of *Listener to NotificationBrokerService. If the notification publisher uses a different API type, MD-SAL automatically translates data in the background.
* REST APIs: Notifications are currently not supported.

==== Instance Identifier

The Instance Identifier is the unique identifier of an element (location) in the yang data tree: basically, it is the *path* to the node that uniquely identifies all the parent nodes of the node. The unique identification of list elements requires the specification of key values as well.

MD-SAL currently provides three different APIs to access data in the common data store: +

* Binding APIs (Java generated DTOs)
* DOM APIs
* https://wiki.opendaylight.org/view/OpenDaylight_Controller:MD-SAL:Restconf[OpenDaylight Controller:MD-SAL Restconf APIs]

*Example* +

Consider the following simple YANG model for inventory: +
----
module inventory {
    namespace "urn:opendaylight:inventory";
    prefix inv;
    revision "2013-06-07";
    container nodes {
        list node {
            key "id";
            leaf "id" {
                type "string";
            }
        }
    }
}
----
*An example having one instance of node with the name _foo_* +

Let us assume that we want to create an instance identifier for the node foo in the following bindings or formats: +


*  **YANG / XML / XPath version**
----
/inv:nodes/inv:node[id="foo"]
----
* **Binding-Aware version (generated APIs)**
----
import org.opendaylight.yang.gen.urn.opendaylight.inventory.rev130607.Nodes;
import org.opendaylight.yang.gen.urn.opendaylight.inventory.rev130607.nodes.Node;
import org.opendaylight.yang.gen.urn.opendaylight.inventory.rev130607.nodes.NodeKey;

import org.opendaylight.yangtools.yang.binding.InstanceIdentifier;

InstanceIdentifier<Node> identifier = InstanceIdentifier.builder(Nodes.class).child(Node.class,new NodeKey("foo")).toInstance();
----
NOTE: The last call, _toInstance()_ does not return an instance of the node, but the Java version of Instance identifier which uniquely identifies the node *foo*.

* **HTTP Restconf APIs** +
----
http://localhost:8080/restconf/config/inventory:nodes/node/foo
----
NOTE: We assume that HTTP APIs are exposed on localhost, port 8080.

* **Binding Independent version (yang-data-api)**
----
import org.opendaylight.yang.common.QName;
import org.opendaylight.yang.data.api.InstanceIdentifier;

QName nodes = QName.create("urn:opendaylight:inventory","2013-06-07","nodes");
QName node = QName.create(nodes,"nodes");
QName idName = QName.create(nodes,"id");
InstanceIdentifier = InstanceIdentifier.builder()
    .node(nodes)
    .nodeWithKey(node,idName,"foo")
    .toInstance();
----
NOTE: The last call, _toInstance()_ does not return an instance of node, but the Java version of Instance identifier which uniquely identifies the node *foo*.

=== MD-SAL: Plugin types
MD-SAL has four component-types that differ in complexity, expose different models, and use different subsets of the MD-SAL functionality.

* Southbound Protocol Plugin: Responsible for handling multiple sessions to the southbound network devices and providing common abstracted interface to access various type of functionality provided by these network devices
* Manager-type application: Responsible for managing the state and the configuration of a particular functionality which is exposed by southbound protocol plugins
* Protocol Library: Responsible for handling serialization or de-serialization between the wire protocol format and the Java form of the protocol
* Connector Plugin: Responsible for connecting consumers (and providers) to Model-driven SAL (and other components) by means of different wire protocol or set of APIs

==== Southbound protocol plugin

The responsibilities of the Southbound Protocol plugin include the following :

* Handling multiple sessions to southbound network devices
* Providing a common abstracted interface to access various type of functionality provided by the network devices

The Southbound Protocol Plugin should be stateless. The only preserved state (which is still transient) is the list of connected devices or sessions. Models mostly use RPCs and Notifications to describe plugin functionality
Example plugins: Openflow Southbound Plugin, Netconf Southbound Plugin, BGP Southbound Plugin, and PCEP Southbound Plugin.

==== Manager-type application

The responsibilities of the Manager-type applications include the following:

* Providing configuration-like functionality to set or modify the behaviour of network elements or southbound plugins
* Coordinating flows and provide higher logic on top of stateless southbound plugins

Manager-type Applications preserve state. Models mostly use Configuration Data and Runtime Data to describe component functionality.

=== Protocol library
The OpenFlow Protocol Library is a component in OpenDaylight, that mediates communication between the OpenDaylight controller and the hardware devices supporting the OpenFlow protocol. The primary goal of the library is to provide user (or upper layers of OpenDaylight) communication channel, that can be used for managing network hardware devices.

=== MD-SAL: Southbound plugin development guide
The southbound controller plugin is a functional component.

The plugin: +

* Provides an abstraction of network devices functionality
* Normalizes their APIs to common contracts
* Handles session and connections to them

The plugin development process generally moves through the following phases: +

. Definition of YANG models (API contracts): For Model-Driven SAL, the API contracts are defined by YANG models and the Java interfaces generated for these models. A developers opts for one of the following: +
** Selects from existing models
** Creates new models
** Augments (extends) existing models
[start=2]
. Code Generation: The Java Interfaces, implementation of Transfer Objects, and mapping to Binding-Independent form is generated for the plugin. This phase requires the proper configuration of the Maven build and YANG Maven Tools.
. Implementation of plugin: The actual implementation of the plugin functionality and plugin components.

NOTE: The order of steps is not definitive, and it is up to the developer to find the most suitable workflow. For additional information, see <<_best_practices>>.

=== Definition of YANG models

In this phase, the developer selects from existing models (provided by controller or other plugins), writes new models, or augments existing ones. A partial list of available models could be found at:
https://wiki.opendaylight.org/view/YANG_Tools:Available_Models[YANG Tools:Available Models].

The mapping of YANG to Java is documented at: https://wiki.opendaylight.org/view/Yang_Tools:YANG_to_Java_Mapping[Yang Tools:YANG to Java Mapping.] This mapping provides an overview of how YANG is mapped to Java.

Multiple approaches to model the functionality of the southbound plugin are available: +

* Using RPCs and Notifications
* Using Configuration Data Description
* Using Runtime Data Description
* Combining approaches

=== RPCs

RPCs can model the functionality invoked by consumers (applications) that use the southbound plugin. Although RPCs can model any functionality, they are usually used to model functionality that cannot be abstracted as configuration data, for example, PacketOut, or initiating a new session to a device (controller-to-device session).

RPCs are modeled with an RPC statement in the following form: +
+rpc foo {}+ +
This statement is mapped to method. +

*RPC input* +
To define RPC input, use an input statement inside RPC. The structure of the input is defined with the same statements as the structure of notifications, configuration, and so on.
----
 rpc foo {
    input {
       ...
    }
 }
----
*RPC output* +
To define the RPC output (structure of result), use the RPC output statement. +
----
 rpc foo {
   output {
      ...
   }
 }
----
*Notifications* +
Use notifications to model events originating in a network device or southbound plugin which is exposed to consumers to listen.


A notification statement defines a notification:
----
   notification foo {
      ...
   }
----
*Configuration data* +

Configuration data is good for the following purposes: +

* Model or provide CRUD access to the state of protocol plugin and/or network devices
* Model any functionality which could be exposed as a configuration to the consumers or applications

Configuration data in YANG is defined by using the config substatement with a true argument. For example: +
----
  container foo {
     config true;
     ...
  }
----
*Runtime (read-only) data* +
Runtime (read-only) data is good to model or provide read access to the state of the protocol plugin and networtk devices, or network devices. This type of data is good to model statistics or any state data, which cannot be modified by the consumers (applications), but needs exposure (for example, learned topology, or list of connected switches).

Runtime data in YANG is defined by using config subsatement with a false argument:
----
  container foo {
     config false;
  }
----
*Structural elements* +
The structure of RPCs, notifications, configuration data, and runtime data is modelled using structural elements (data schema nodes). Structural elements define the actual structure of XML, DataDOM documents, and Java APIs for accessing or storing these elements. The most commonly used structural elements are: +

* Container
* List
* Leaf
* Leaf-list
* Choice

=== Augmentations +
Augmentations are used to extend existing models by providing additional structural elements and semantics. Augmentation cannot change the mandatory status of nodes in the original model, or introduce any new mandatory statements.

=== Best practices

* YANG models must be located under the src/main/yang folder in your project.
* Design your models so that they are reusable and extendible by third-parties.
* Always try to reuse existing models and types provided by these models. See https://wiki.opendaylight.org/view/YANG_Tools:Available_Models[YANG Tools:Available Models] or others if there is no model which provides you with data structures and types you need.

*Code generation* +
To configure your project for code generation, your build system needs to use Maven. For the configuration of java API generation, see https://wiki.opendaylight.org/view/Yang_Tools:Maven_Plugin_Guide[Yang Tools:Maven Plugin Guide].

*Artefacts generated at compile time* +
The following artefacts are generated at compile time: +

* Service interfaces
* Transfer object interfaces
* Builders for transfer objects and immutable versions of transfer objects

=== Implementation +
This step uses generated artefacts to implement the intended functionality of the southbound plugin. +

*Provider implementation* +
To expose functionality through binding-awareness, the MD-SAL plugin needs to be compiled against these APIs, and must at least implement the BindingAwareProvider interface.
The provider uses APIs which are available in the SAL-binding-api Maven artifact. To use this dependency, insert the following dependency into your pom.xml:
----
<dependency>
       <groupId>org.opendaylight.controller</groupId>
       <artifactId>sal-binding-api</artifactId>
       <version>1.0-SNAPSHOT</version>
   </dependency>
----

*BindingAwareProvider implementation* +
A BindingAwareProvider interface requires the implementation of four methods, and registering an instance with BindingAwareBroker. Use AbstractBindingAwareProvider to simplify the implementation.

* void onSessionInitialized(ConsumerContext ctx): This callback is called when Binding-Aware Provider is initialized and ConsumerContext is injected into it. ConsumerContext serves to access all functionality which the plugin is to consume from other controller components.
* void onSessionInitialized(ProviderContext ctx): This callback is called when Binding-Aware Provider is initialized and ProviderContext is injected into it. ProviderContext serves to access all functionality which the plugin could use to provide its functionality to controller components.
* Collection<? extends RpcService> getImplementations(): Shorthand registration of an already instantiated implementations of global RPC services. Automated registration is currently not supported.
* public Collection<? extends ProviderFunctionality> getFunctionality(): Shorthand registration of an already instantiated implementations of ProviderFunctionality. Automated registration is currently not supported.
NOTE: You also need to set your implementation of AbstractBindingAwareProvider set as Bundle Activator for MD-SAL to properly load it.

=== Notifications
To publish events, request an instance of NotificationProviderService from ProviderContext. Use the following:
----
   ExampleNotification notification = (new ExampleNotificationBuilder()).build();
   NotificationProviderService notificationProvider = providerContext.getSALService(NotificationProviderService.class);
   notificationProvider.notify(notification);
----
*RPC implementations* +
To implement the functionality exposed as RPCs, implement the generated RpcService interface. Register the implementation within ProviderContext included in the provider.

If the generated RpcInterface is FooService, and the implementation is FooServiceImpl:
----
   @Override
   public void onSessionInitiated(ProviderContext context) {
       context.addRpcImplementation(FooService.class, new FooServiceImpl());
   }
----
=== Best practices

RPC Service interface contract requires you to return http://docs.oracle.com/javase/7/docs/api/java/util/concurrent/Future.html[Future object] (to make it obvious that call may be asynchronous), but it is not specified how this Future is implemented. Consider using existing implementations provided by JDK or Google Guava. Implement your own Future only if necessary.

Consider using http://docs.guava-libraries.googlecode.com/git-history/release/javadoc/com/google/common/util/concurrent/SettableFuture.html[SettableFuture] if you intend not to use http://docs.oracle.com/javase/7/docs/api/java/util/concurrent/FutureTask.html[FutureTask] or submit http://docs.oracle.com/javase/7/docs/api/java/util/concurrent/Callable.html[Callables] to http://docs.oracle.com/javase/7/docs/api/java/util/concurrent/ExecutorService.html[ExecutorService].

IMPORTANT: Do not implement transfer object interfaces unless necessary. Choose already generated builders and immutable versions. If you want to implement transfer objects, ensure that instances exposed outside the plugin are immutable.

=== OpenDaylight Controller: MD-SAL FAQs

*Q-1: What is the overall MD-SAL architecture?*

* **What is the overall architecture, components, and functionality?**
* **Who supplies which components, and how are the components plumbed?**

*A-1:* The overall Model-Driven SAL (MD-SAL) architecture did not really change from the API-Driven SAL (AD-SAL). As with the AD-SAL, plugins can be data providers, or data consumers, or both (although the AD-SAL did not explicitly name them as such). Just like the AD-SAL, the MD-SAL connects data consumers to appropriate data providers and (optionally) facilitates data adaptation between them.

Now, in the AD-SAL, the SAL APIs request routing between consumers and providers, and data adaptations are all statically defined at compile or build time. In the MD-SAL, the SAL APIs and request routing between consumers and providers are defined from models, and data adaptations are provided by 'internal' adaptation plugins. The API code is generated from models when a plugin is compiled. When the plugin OSGI bundle is loaded into the controller, the API code is loaded into the controller along with the rest of the plugin containing the model.

.AD-SAL and MD-SAL
image::MD-SAL.png[]

The AD-SAL provides request routing (selects an SB plugin based on service type) and optionally provides service adaptation, if an NB (Service, abstract) API is different from its corresponding SB (protocol) API. For example, in the above figure, the AD-SAL routes requests from NB-Plugin 1 to SB Plugins 1 and 2. Note that the plugin SB and NB APIs in this example are essentially the same (although both of them need to be defined). Request routing is based on plugin type: the SAL knows which node instance is served by which plugin. When an NB Plugin requests an operation on a given node, the request is routed to the appropriate plugin which then routes the request to the appropriate node. The AD-SAL can also provide service abstractions and adaptations. For example, in the above figure, NB Plugin 2 is using an abstract API to access the services provided by SB Plugins 1 and 2. The translation between the SB Plugin API and the abstract NB API is done in the Abstraction module in the AD-SAL.

The MD-SAL provides request routing and the infrastructure to support service adaptation. However, it does not provide service adaptation itself: service adaptation is provided by plugins. From the point of view of MD-SAL, the Adaptation Plugin is a regular plugin. It provides data to the SAL, and consumes data from the SAL through APIs generated from models. An Adaptation Plugin basically performs model-to-model translations between two APIs. Request Routing in the MD-SAL is done on both protocol type and node instances, since node instance data is exported from the plugin into the SAL (the model data contains routing information).

The simplest MD-SAL APIs generated from models (RPCs and Notifications, both supported in the yang modeling language) are functionally equivalent to AD-SAL function call APIs. Additionally, the MD-SAL can store data for models defined by plugins. Provider and consumer plugins can exchange data through the MD-SAL storage. Data in the MD-SAL is accessed through getter and setter APIs generated from models. Note that this is in contrast to the AD-SAL, which is stateless.

Note that in the above figure, both NB AD-SAL Plugins provide REST APIs to controller client applications.

The functionality provided by the MD-SAL is basically to facilitate the plumbing between providers and consumers. A provider or a consumer can register itself with the MD-SAL. A consumer can find a provider that it is interested in. A provider can generate notifications; a consumer can receive notifications and issue RPCs to get data from providers. A provider can insert data into SAL storage; a consumer can read data from SAL storage.

Note that the structure of SAL APIs is different in the MD-SAL from that in the AD-SAL. The AD-SAL typically has both NB and SB APIs even for functions or services that are mapped 1:1 between SB Plugins and NB Plugins. For example, in the current AD-SAL implementation of the OpenFlow Plugin and applications, the NB SAL APIs used by OF applications are mapped 1:1 onto SB OF Plugin APIs. The MD-SAL allows both the NB plugins and SB plugins to use the same API generated from a model. One plugin becomes an API (service) provider; the other becomes an API (service) Consumer. This eliminates the need to define two different APIs and to provide three different implementations even for cases where APIs are mapped to each other 1:1. The MD SAL provides instance-based request routing between multiple provider plugins.

*Q-2: What functionality does the MD-SAL assume? For example, does the SAL assume that the network model is a part of the SAL?*

*A-2:* The MD-SAL does not assume any model. All models are provided by plugins. The MD-SAL only provides the infrastructure and the plumbing for the plugins.


*Q-3: What is the "day in the life" of an MD-SAL plugin?*


*A-3:* All plugins (protocol, application, adaptation, and others) have the same lifecycle. The life of a plugin has two distinct phases: design and operation. +
During the design phase, the plugin designer performs the following actions:  +

* The designer decides which data will be consumed by the plugin, and imports the SAL APIs generated from the API provider’s models. Note that the topology model is just one possible data type that may be consumed by a plugin. The list of currently available data models and their APIs can be found in YANG_Tools:Available_Models.
* The designer decides which data and how it will be provided by the plugin, and designs the data model for the provided data. The data model (expressed in yang) is then run through the https://wiki.opendaylight.org/view/YANG_Tools:Available_Models[YANG Tools], which generate the SAL APIs for the model.
* The implementations for the generated consumer and provider APIs, along with other plugin features and functionality, are developed. The resulting code is packaged in a “plugin” OSGI bundle. Note that a developer may package the code of a subsystem in multiple plugins or applications that may communicate with each other through the SAL.
* The generated APIs and a set of helper classes are also built and packaged in an “API” OSGI bundle.

The plugin development process is shown in the following figure. +

.Plugin development process
image::plugin-dev-process.png[]

When the OSGI bundle of a plugin is loaded into the controller and activated, the operation phase begins. The plugin operation is probably best explained with a few examples describing the operation of the OF Protocol plugin and OF applications, such as the Flow Programmer Service, the ARP Handler, or the Topology Manager. The following figure shows a scenario where a “Flow Deleted” notification from a switch arrives at the controller.

.Flow deleted at controller
image::flow-deleted-at-controller.png[]

The scenario is as follows: +

. The Flow Programmer Service registers with the MD SAL for the `Flow Deleted' notification. This is done when the Controller and its plugins or applications are started.
. A `Flow Deleted' OF packet arrives at the controller. The OF Library receives the packet on the TCP/TLS connection to the sending switch, and passes it to the OF Plugin.
. The OF Plugin parses the packet, and uses the parsed data to create a `Flow Deleted' SAL notification. The notification is actually an immutable `Flow Deleted' Data Transfer Object (DTO) that is created or populated by means of methods from the model-generated OF Plugin API.
. The OF Plugin sends the `Flow Deleted' SAL notification (containing the notification DTO) into the SAL. The SAL routes the notification to registered consumers, in this case, the Flow Programmer Service.
. The Flow Programmer Service receives the notification containing the notification DTO.
. The Flow Programmer Service uses methods from the API of the model-generated OF Plugin to get data from the immutable notification DTO received in Step 5. The processing is the same as in the AD-SAL.

Note that other packet-in scenarios, where a switch punts a packet to the controller, such as an ARP or an LLDP packet, are similar. Interested applications register for the respective notifications. The OF plugin generates the notification from received OF packets, and sends them to the SAL. The SAL routes the notifications to the registered recipients. +
The following figure shows a scenario where an external application adds a flow by means of the NB REST API of the controller.

.External app adds flow
image::md-sal-faqs-add_flow.png[]

The scenario is as follows: +

. Registrations are performed when the Controller and its plugins or applications are started.

.. The Flow Programmer Service registers with the MD SAL for Flow configuration data notifications.
.. The OF Plugin registers (among others) the ‘AddFlow’ RPC implementation with the SAL.
Note that the RPC is defined in the OF Plugin model, and the API is generated during build time. +
[start=2]
. A client application requests a flow add through the REST API of the Controller. (Note that in the AD-SAL, there is a dedicated NB REST API on top of the Flow Programming Service. The MD-SAL provides a common infrastructure where data and functions defined in models can be accessed by means of a common REST API. For more information, see http://datatracker.ietf.org/doc/draft-bierman-netconf-restconf/). The client application provides all parameters for the flow in the REST call.
. Data from the ‘Add Flow’ request is deserialized, and a new flow is created in the Flow Service configuration data tree. (Note that in this example the configuration and operational data trees are separated; this may be different for other services). Note also that the REST call returns success to the caller as soon as the flow data is written to the configuration data tree.
. Since the Flow Programmer Service is registered to receive notifications for data changes in the Flow Service data tree, the MD-SAL generates a ‘data changed’ notification to the Flow Programmer Service.
. The Flow Programmer Service reads the newly added flow, and performs a flow add operation (which is basically the same as in the AD-SAL).
. At some point during the flow addition operation, the Flow Programmer Service needs to tell the OF Plugin to add the flow in the appropriate switch. The Flow Programmer Service uses the OF Plugin generated API to create the RPC input parameter DTO for the “AddFlow” RPC of the OF Plugin.
. The Flow Programmer Service gets the service instance (actually, a proxy), and invokes the “AddFlow” RPC on the service. The MD-SAL will route the request to the appropriate OF Plugin (which implements the requested RPC).
. The `AddFlow' RPC request is routed to the OF Plugin, and the implementation method of the “AddFlow” RPC is invoked.
. The `AddFlow' RPC implementation uses the OF Plugin API to read values from the DTO of the RPC input parameter. (Note that the implementation will use the getter methods of the DTO generated from the yang model of the RPC to read the values from the received DTO.)
. The `AddFlow' RPC is further processed (pretty much the same as in the AD-SAL) and at some point, the corresponding flowmod is sent to the corresponding switch.

*Q-4: Is there a document that describes how code is generated from the models for the MD-SAL?*

*A-4:* https://wiki.opendaylight.org/view/YANG_Tools:YANG_to_Java_Mapping[Yangtools] documents the Yang to Java generation, including examples of how the yang constructs are mapped into Java classes. You can write unit tests against the generated code. You will have to write implementations of the generated RPC interfaces. The generated code is just Java, and it debugs just like Java.

If you want to play with generating Java from Yang there is a maven archetype to help you get going: https://wiki.opendaylight.org/view/Maven_Archetypes:odl-model-project[Maven Archetypes: ODL Model Project]. +
Or, you can try creating a project in Eclipse as explained at: http://sdntutorials.com/yang-to-java-conversion-how-to-create-maven-project-in-eclipse/[YANG to Java conversion: How to create Maven project in Eclipse].

*Q-5: The code generation tools mention 'producers' and consumers'. How are these related to 'southbound' and 'northbound SAL plugins?*

*A-5:* The difference between southbound and northbound plugins is that the southbound plugins talk protocols to network nodes, and northbound plugins talk application APIs to the controller applications. As far as the SAL is concerned, there is really no north or south. The SAL is basically a data exchange and adaptation mechanism between plugins. The plugin SAL roles (consumer or producer) are defined with respect to the data being moved around or stored by the SAL. A producer implements an API, and provides the data of the API: a consumer uses the API, and consumes the data of the API. +
While 'northbound' and 'southbound' provide a network engineer's view of the SAL, 'consumer' and 'producer' provide a software engineer's view of the SAL, and is shown in the following figure:

.SAL consumer and producer view

image::mdsal-sal-sw-eng.png[]

*Q-6: Where can I find models that have already been defined in OpenDaylight?*

*A-6:* The list of models that have been defined for the SAL and in various plugins can be found in https://wiki.opendaylight.org/view/OpenDaylight_Controller:MD-SAL:Model_Reference[MD-SAL Model Reference].

*Q-7: How do I migrate my existing plugins and services to MD-SAL?*

*A-7:* The migration guide can be found in the https://wiki.opendaylight.org/view/OpenDaylight_Controller:MD-SAL:Application_Migration_Guide[MD-SAL Application Migration Guide].

*Q-8: Where can I find SAL example code?*

*A-8:* The toaster sample provides a simple yet complete example of a model, a service provider (toaster), and a service consumer. It provides the model of a programmable toaster, a sample consumer application that uses MD-SAL APIs; a sample southbound plugin (a service provider) that implements toaster; and a unit test suite.

The toaster example is in _controller.git_ under _opendaylight/md-sal/samples_.

*Q-9: Where is the REST API code for the example?*

*A-9:* The REST APIs are derived from models. You do not have to write any code for it. The controller will implement the http://datatracker.ietf.org/doc/draft-bierman-netconf-restconf/[RESTCONF protocol] which defines access to yang-formatted data through REST. Basically, all you need to do is define your service in a model, and expose that model to the SAL. REST access to your modeled data will then be provided by the SAL infrastructure. However, if you want to, you can create your own REST API (for example, to be compliant with an existing API).

*Q-10: How can one use RESTCONF to access the MD-SAL datastore?*

*A-10:* For information on accessing the MD-SAL datastore, see https://wiki.opendaylight.org/view/OpenDaylight_Controller:MD-SAL:Restconf[MD-SAL Restconf].



=== MD-SAL architecture: Clustering Notifications
MD-SAL supports two kinds of messaging exchange pattern: +

* Request/Reply
* Publish/Subscribe
The RPC module implements the Request/Reply pattern. The notification module implements the Publish/Subscribe functionality. The implementation details are provided at: https://wiki.opendaylight.org/view/OpenDaylight_Controller:MD-SAL:Explained:Messaging_Patterns[OpenDaylight Controller:MD-SAL:Explained:Messaging Patterns].
The focus now is on Publish/Subscribe implementation.An earlier implementation assumed a single VM deployment of the controller.The message exchange happens only within a VM in memory. The current requirement is to enable these notifications across nodes in the cluster.

Publish/Subscribe notifications are of two kinds: +

* Data Change events
* Yang notifications
In both cases, the notifications are broadcast to all "listeners". +
*Requirements* +
Some of the requirements: +

* Ability to publish notifications to any subscriber in the cluster
* Subscriber ability to specify delivery policy
* 1 of N: Delivery of the notification to any one of N instances of application running in the cluster
* N of N: Broadcasts
* Local only: Notifying events generated on the same node as the application instance
* Load Balancing: Round robin, least loaded etc
* Content Based or any other application specified custom logic
* Publisher capability to attach properties to the message
* Message priority
* Delivery guarantee
* Ability to plug-in external systems such as AMQP based systems

==== Proposed change
Based on the requirements, a change in the aPI was proposed: +
----
 Yang notification
 publish(Notification notification, MessageProperties props);
 registerNotificationListener(org.opendaylight.yangtools.yang.binding.NotificationListener.NotificationListener listener, Selector selector);
 registerNotificationListener(Class notificationType, org.opendaylight.controller.sal.binding.api.NotificationListener listener, Selector selector);
 Data change notification
 registerDataChangeListener(LogicalDatastoreType store, P path, L listener, DataChangeScope triggeringScope, "Selector selector");
public interface MessageProperties{
 public Priority priority();
 ...[add more properties]
}
public enum Priority { HIGH, NORMAL, LOW};
public interface Selector {
 public List<InstanceLocator> select(Notification event, List<InstanceLocator> instances);
}
----

=== MD-SAL Architecture: DOM
There are several issues that impede the reliability and performance of mD-SAL: +

* Data structures (defined in yang-data-api) are like XML structures. Therefore, it is hard to implement an optimized datastore atop them. Instead, YANG-defined data structures must be used in the data store. YANG-defined data structures are already being used in the MD-SAL: in the Java DTOs generated by YangTools, and in other components.
* The current MD-SAL data contracts do not provide enough capabilities to more accurately specify an the intent of an application and to perform optimizations to clients (for example, 'do not unnecessarily deserialize data', or 'compute only necessary change sets'). The current datastore implementation prevents atomic updates on subtrees.

==== MD-SAL DOM Data Broker
The current DOM Data Broker design does not include an assumption of a intelligent in-memory cache with tree-like structures that would:

* Be able to track dependencies
* Calculate change sets
* Maintain the relationships between commit handlers, notification listeners and the actual data.
This may lead to an inefficient implementation of the two-phase commit, where all state tracking during the is done by the Data Broker itself as follows: +
. Calculate the affected subtrees.
. Filter the commit handlers by the affected subtrees.
. Filter data change listeners by the affected subtrees.
. Capture the initial state for data change listeners (one read per data change listener set).
. Start Request Commit of all the affected commit handlers.
. Finish Commit on all the affected commit handlers.
. Capture the final state for data change listeners (one read per data change listener set).
. Publish the Data Change events to the affected data change listeners.
The states that the current DOM Data Broke keeps and maintains are mapping of subtree paths to:  *

* Registered commit handlers
* Registered data change listeners
* Registered data readers
DOM Data Broker has the following state keeping responsibilities: *

* Read request routing for data readers
* Two phase commit coordination
* Publish Data Change Events
* Capture Before and After state


// FIXME: Remove this section since it describes PoC which probably has no value to developers.
=== MD-SAL: Infinispan Data Store

==== Components of Infinispan Data Store
Infinispan Data Store comprises the following major components: +

* Encoding or Decoding a Normalized Node into and from the Infinispan TreeCache
* Managing transactions
* Managing DataChange notifications

==== Encoding or Decoding a Normalized Node into and from the Inifinispan TreeCache +
A NormalizedNode represents a tree whose structure closely models the yang model of a bunch of modules. The NormalizedNode tree typically has values either placed in: +

* A LeafNode (corresponding to a leaf in yang)
* A LeafSetEntryNode (corresponding to a leaflist in yang) +
The encoding logic walks the NormalizedNode tree looking for LeafNodes and LeafSetEntryNodes.When the logic finds a LeafNode or a LeafSetEntryNode, it records the finding in a map with the following: +

* Instance Identifier of the parent as the key
* The value of the leaf or leafset entry store in a map where:
** The NodeIdentifier of the leaf/leafsetentry is the key.
** The value of the leaf/leafsetentry is the value.
The decoding process involves the following steps: +

. Uses the interface of TreeCache to get to a certain node in the tree
. Walks through the tree, and reconstructs the NormalizedNode based on the key and value in the Infinispan TreeCache
. Validates the NormalizedNode against the schema

==== Managing Transactions +
To ensure read-write isolation level, and for other reasons, an infinispan (JTA) transaction for each datastore transaction is created. Since a single thread may be used for multiple JTA transactions,
the implementation has to ensure the suspension and resumption of the JTA transactions appropriately.
However, this does not seem to have an impact on performance.

==== Managing DataChange notifications +
The current interface for data change notifications supports the registering of listeners for the following notifications: +

* Data changes at Node (consider node of a tree) level
* Events for any changes that happen at *one* level (meaning immediate children)
* Any change at the subtree level
The event sent to the listener requires that the following snapshots of the tree be maintained: +

* Before data change
* After data change

NOTE: This process is very expensive. It means maintaining a Normalized Node representing a snapshot of the tree. It involves converting the tree in Infinispan to NormalizeNode object tree required by the consumer at the start of each transaction.

*To maintain the data changes:* +

. At the begin of transaction, get a NormalizedNode Object tree of the current tree in ISPN TreeCache (This is mandated by the current DataChangeEvent interface.)
. For each CUD operations that happens within the transaction, maintain a transaction log.
. When the pre-commit of the 3PhaseCommit Transaction Interface is called, prepare data changes. This involves: +
.. Comparing the transaction log items with the Snapshot Tree one taken at the beginning of the transactions
.. Preparing the DataChangeEvent lists based on what level the listeners have registered
. Upon a commit, send the events to the listeners in a separate executor, that is asynchronously.

*Suggested changes* +

* Remove the requirement for sending the `before transaction tree' or the `after transaction tree' within each event.
* Send the changed paths of tree to the consumer, and let the consumer do the reading.

==== Building the POC +
To build or run the POC, you need the latest version of the following: +

* Yangtools
* Controller
* OpenFlow plugin

==== To get yangtools +

. Get the latest yangtools sources, and then create a branch of it using the following command:
: git checkout 306ffd9eea5a52556b4877debd2a79ca0573ff0c -b infinispan-data-store +
. Build using the following command:
: mvn clean install -DskipTests +

==== To get the Controller

. Get the latest controller, and then create a branch using the following command:
: git checkout 259b65622b8c29c49235c2210609b9f7a68826eb -b infinispan-data-store +
. Apply the following gerrit.
: https://git.opendaylight.org/gerrit/#/c/5900/
. Build using the following command:
: mvn clean install -DskipTests +
. If the build should fails, use the following commang:
: cd opendaylight/md-sal/sal-ispn-datastore +
. Build using the following command:
+mvn clean install+
. Return to the controller directory, and build using:
: mvn clean install -DskipTests or resume build +

==== To get the OpenFlowplugin

. Get the latest openflowplugin code and then create a branch using the following command:
: git checkout 6affeefef4de51ce4b7de86fd9ccf51add3922f7 -b infinispan-data-store +
. Build using the following command:
: mvn clean install -DskipTests +
. Copy the sal-ispn-datastore jar to the plugins folder.

==== Running the POC +
*Prerequisite* +
Ensure that the 01-md-sal.xml file has been changed to use the new MD-SAL datastore. +

* Run the controller with the infinispan datastore. The section, <<_comparison_of_in_memory_and_infinispan_datastore>> provides information about cbench testing.

NOTE: If you want to see performance numbers similar to those documented, disable datachange notifications.
The only way to do that in the POC is to change the code in ReadWriteTransactionImpl. Look for the FIXME comments.

=== State of the POC +

* Encoding and Decoding a Normalized Node into an Infinispan TreeCache works
* Integrated with the controller
* Eventing works
* With Data Change events disabled, the Infinispan based datastore performs the same, or better than, the custom In-Memory Datastore. Although initially slow, with time it seems to perform more consistently than the In-Memory Datastore.,
* Not fully tested

=== Infinispan-related learnings +

*Below par functioning of TreeCache#removeNode API* +
The Infinispan removeNode API failed to remove nodes in the tree, as was promised, correctly. This means, for example, that when a mininet topology changes, some nodes may not be removed from inventory and topology.
This behaviour has not been properly evaluated, and no remedy is currently available.

=== Datastore-related learnings +

*Multiple transactions can be created per thread* +
This is a problem because if the backing datastore (infinispan) uses JTA transactions, only one transaction can be active per thread.
Although this does not necessarily mean the usage of one thread per transaction, it calls for the suspension of one transaction and the resumption of another.
TIP::
* Allow only one active transaction per thread.
* Add an explicit suspend or resume method to a transaction.

=== No clarity on the closing of Read-Only transactions +
For every DataStore transaction, a JTA transaction needs to be created. This is to ensure isolation (repeatable reads). When the transaction is done, it must be committed, rolled back, or closed in some fashion. Read-only transactions may not close. This leads to JTA transactions being open until they are timed out.

TIP::

* A DataStore may need to do time-outs as well.
* Call _close_ explicitly for read-only transactions.

==== Write and Delete methods in a read-write transaction do not return a Future
The Write and Delete methods on the DOMWriteTransaction return a void instead of a Future, creating the impression that these methods are synchronous. This is not necessarily true in all cases: for example, in the infinispan datastore, the write was actually done in a separate thread to support multiple transactions on a single thread.
TIP: Return a ListenableFuture for both Write and Delete methods.

==== Expense of creating a DataChange event +
Creating a DataChange event is very expensive because it needs to pass the Original Sub tree and the Modified Sub tree. +
A NormalizedNode object needs to be created to create a DataChange event. The NormalizedNode object may be a snapshot of the complete modules data to facilitate the sending of the original subtree to DataChange listeners. The prohibitive expense prevents this implementation in every transaction. This is a problem not only in the infinispan datastore but also in a distributed system. A distributed system shards data to collocate it on a different node on the cluster with applications and datachange listeners. For example, while a system may have shards collocated with the inventory application; the topology application may be a datachange listener for datachange events. In this case, the original subtree and the modified sub tree would need to be serialized in some form, and sent to the topology listener.
TIP: Remove the getOriginalSubtree and getModifiedSubtree methods from the datachange listener; understand the use case for providing them; and find a cheaper alternative.

==== Complications of reconstructing a Normalized Node from different data-structures +
The reconstruction of a Normalized Node from a different data-structure, like a map or a key-value store, is complicated or may appear complicated.
A NormalizedNode is the binding-independent equivalent of data that gets stored in the datastore. For the in-memory datastore, it is the native storage format. It is a complicated structure that basically mirrors the model as defined in yang. Understanding it and properly decoding it could be a challenge for the implemention of an alternate datastore.
TIP: Create utility classes to construct a normalized node from a simple tree structure. The Old CompositeNode or the Infinispan Node for example is a much simpler structure to follow.

==== Comparison of In-Memory and Infinispan Datastore
Cbench was used to compare the performance of the two datastores.
To prepare the controller for testing: +

IMPORTANT: Use the openflow plugin distribution.

. Remove the simple forwarding, arp handler, and md-sal statistics manager bundles.
. Set the log level to ERROR.
. Run the controller with the following command: +
:  ./run.sh -Xmx4G -Xms2G -XX:NewRatio=5 -XX:+UseG1GC -XX:MaxPermSize=256m
. From the osgi command prompt, use *dropAllPackets on*.

==== Running cbench +
For both the in-memory and infinispan datastore versions, cbench was run 11 times. The first run is ignored in both cases.

* Use the cbench command: +
: cbench -c <controller ip> -p 6633 -m 1000 -l 10 -s 16 -M 1000
This was a latency test and the arguments roughly translate to this: +
: -m 1000 : use 1000 milliseconds per test -l 10 : use 10 loops per test -s 16 : fake 16 switches -M 1000 : use 1000 hosts per switch
 </div>

==== The results for In-Memory Datastore +
To test the in-memory datastore, a pre-built openflow plugin distribution from Jenkins was downloadedon and on which was enabled the new in-memory datastore. +
*In-Memory Datastore Results*
[options="header",width="75%"]
|===
| Run | Min | Max | Avg | StdDev
| 1 | 365 | 1049 | 715 | 04
| 2 | 799 | 1044 | 953 | 71
| 3 | 762 | 949 | 855 | 59
| 4 | 616 | 707 | 666 | 27
| 5 | 557 | 639 | 595 | 24
| 6 | 510 | 583 | 537 | 25
| 7 | 455 | 535 | 489 | 22
| 8 | 351 | 458 | 420 | 38
| 9 | 396 | 440 | 417 | 14
| 10 | 376 | 413 | 392 | 13
|===

==== Infinispan Datastore +
The Infinispan Datastore was built of a master a month old. Since the In-Memory datastore was hardcoded at that time the in-memory datastore was swapped for the the infinispan datastore by modifying the sal-broker-impl sources.


Listed are some steps that were either completed to isolate the changes that were being made, or to tweak performance:  +

* Infinispan 5.3 was used because to isolate changes to utilize tree cache to the infinispan datastore bundles. Attempting to use version 6.0 caused a problem in loading some classes from infinispan.Ideally, to use infinispan as a backing store, tweak clustering services to obtain a treecache.
* Added an exists method onto the In-Memory ReadTransaction API. This was because it was found that in one place in the BA Broker was code which checked for the existence of nodes in the tree by doing a read. Reads are a little expensive on the Infinispan datastore because of the need to convert to a NormalizedNode. An exists method was added to the interface to just check for node-existence.
* When a transaction was used to read data it was not being closed causing the Infinispan JTA transactions to persist. Again, a change in the broker was made to close a transaction after it was concluded so that it dis not persist and trigger a clean by the reaper.

*Infinispan Datastore Results*
[cols="5*",^,options="header",width="75%"]
|===
| Run | Min | Max | Avg | StdDev
| 1 | 43 | 250 | 186 | 61
| 2 | 266 | 308 | 285 | 13
| 3 | 300 | 350 | 325 | 12
| 4 | 378 | 446 | 412 | 24
| 5 | 609 | 683 | 644 | 26
| 6 | 492 | 757 | 663 | 76
| 7 | 794 | 838 | 816 | 11
| 8 | 645 | 845 | 750 | 60
| 9 | 553 | 829 | 708 | 100
| 10 | 615 | 910 | 710 | 86
|===