Add basic PingPongTransactionChainTest

[mdsal.git] / docs / conceptual-data-tree.rst

####################
Conceptual Data Tree
####################

.. _terminology:

Terminology
===========

:Data Tree:
    An instantiated logical tree that represents configuration or operational
    state data of a modeled problem domain (for example, a controller or a
    network)

:Data Tree Consumer:
    A component acting on data, after this data are introduced into one or more
    particular subtrees of a Data Tree.

:Data Tree Identifier:
    A unique identifier for a particular subtree of a Data Tree. It is composed
    of the logical data store type and the instance identifier of the subtree's
    root node. It is represented by a ``DOMDataTreeIdentifier``.

:Data Tree Producer:
    A component responsible for providing data for one or more particular
    subtrees of a Data Tree.

:Data Tree Shard:
    A component responsible for providing storage or access to a particular
    subtree of a Data Tree.

:Shard Layout:
    A longest-prefix mapping between Data Tree Identifiers and Data Tree Shards
    responsible for providing access to a data subtree.

Basic Concepts
==============

Data Tree is a Namespace
------------------------

The concept of a data tree comes from `RFC6020
<https://tools.ietf.org/html/rfc6020>`_. It is is vaguely
split into two instances, configuration and operational. The implicit
assumption is that **config implies oper**, i.e. any configuration data is
also a valid operational data. Further interactions between the two are left
undefined and the YANG language is not strictly extensible in the number and
semantics of these instances, leaving a lot to implementation details. An
outline of data tree use, which is consistent with the current MD-SAL design,
is described in `draft-kwatsen-netmod-opstate
<https://tools.ietf.org/html/draft-kwatsen-netmod-opstate>`_.

The OpenDaylight MD-SAL design makes no inherent assumptions about the
relationship between the configuration and operational data tree instances.
They are treated as separate entities and they are both fully addressable via
the ``DOMDataTreeIdentifier`` objects. It is up to MD-SAL plugins (e.g.
protocol plugins or applications) to maintain this relationship. This reflects
the asynchronous nature of applying configuration and also the fact that the
intended configuration data may be subject to translation (such as template
configuration instantiation).

Both the configuration and operational namespaces (data trees) are instances
of the Conceptual Data Tree. Any data item in the conceptual data tree is
addressed via a ``YangInstanceIdentifier`` object, which is a unique,
hierarchical, content-based identifier. All applications use the identifier
objects to identify data to MD-SAL services, which in turn are expected to
perform proper namespace management such that logical operation connectivity is
maintained.

.. todo::

   Can you reword '...are expected to perform proper namespace management such
   that logical operation connectivity is maintained...' - not clear what you
   mean

.. _identifiers-vs-locators:

Identifiers versus Locators
---------------------------

It is important to note that when we talk about Identifiers and Locators,
we **do not** mean `URIs and URLs
<https://en.wikipedia.org/wiki/Uniform_Resource_Identifier>`_,
but rather URNs and URLs as strictly separate entities. MD-SAL plugins do not
have access to locators and it is the job of MD-SAL services to provide
location independence.

The details of how a particular MD-SAL service achieves location independence
is currently left up to the service's implementation, which leads to the
problem of having MD-SAL services cooperate, such as storing data in different
backends (in-memory, SQL, NoSQL, etc.) and providing unified access to all
available data. Note that data availability is subject to capabilities of a
particular storage engine and its operational state, which leads to the design
decision that a ``YangInstanceIdentifier`` lookup needs to be performed in two
steps:

#. A longest-prefix match is performed to locate the storage backend instance
   for that identifier
#. Masked path elements are resolved by the storage engine

.. _data-tree-shard:

Data Tree Shard
---------------

A process similar to the first step above is performed today by the Distributed
Data Store implementation to split data into Shards. The concept of a Shard as
currently implemented is limited to specifying namespaces, and it does not
allow for pluggable storage engines.

In context of the Conceptual Data Tree, the concept of a Shard is generalized
as the shorthand equivalent of a storage backend instance. A Shard can be
attached at any (even wildcard) ``YangInstanceIdentifier``. This contract is
exposed via the ``DOMShardedDataTree``, which is an MD-SAL SPI class that
implements an ``YangInstanceIdentifier`` -> ``Shard`` registry service. This is
an omnipresent MD-SAL service, Shard Registry, whose visibility scope is a
single OpenDaylight instance (i.e. a cluster member). **Shard Layout** refers
to the mapping information contained in this service.

Federation, Replication and High Availability
---------------------------------------------

Support for various multi-node scenarios is a concern outside of core MD-SAL.
If a particular scenario requires the shard layout to be replicated (either
fully or partially), it is up to Shard providers to maintain an omnipresent
service on each node, which in turn is responsible for dynamically registering
``DOMDataTreeShard`` instances with the Shard Registry service.

Since the Shard Layout is strictly local to a particular OpenDaylight instance,
an OpenDaylight cluster is not strictly consistent in its mapping of
``YangInstanceIdentifier`` to data. When a query for the entire data tree is
executed, the returned result will vary between member instances based on the
differences of their Shard Layouts. This allows each node to project its local
operational details, as well as the partitioning of the data set being worked
on based on workload and node availability.

Partial symmetry of the conceptual data tree can still be maintained to
the extent that a particular deployment requires. For example the Shard
containing the OpenFlow topology can be configured to be registered on all
cluster members, leading to queries into that topology returning consistent
results.

.. _design:

Design
======

.. _design-listener:

Data Tree Listener
------------------

A Data Tree Listener is a data consumer, for example a process that wants
to act on data after it has been introduced to the Conceptual Data Tree.

A Data Tree Listener implements the :mdsal-apidoc:`DOMDataTreeListener
<DOMDataTreeListener.html>` interface and registers itself using
:mdsal-apidoc:`DOMDataTreeService <DOMDataTreeService.html>`.

A Data Tree Listener may register for multiple subtrees. Each time it is
invoked it will be provided with the current state of all subtrees that it
is registered for.

.. todo:: Consider linking / inlining interface

.DOMDataTreeListener interface signature

.. code-block:: java

   public interface DOMDataTreeListener extends EventListener {

       void onDataTreeChanged(Collection<DataTreeCandidate> changes, // (1)
           Map<DOMDataTreeIdentifier, NormalizedNode<?, ?>> subtrees);

       void onDataTreeFailed(Collection<DOMDataTreeListeningException> causes); // (2)
   }

#. Invoked when the data tree to which the Data Tree Listener is subscribed
   to changed. `changes` contains the collection of changes, `subtrees`
   contains the current state of all subtrees to which the listener is
   registered.
#. Invoked when a subtree listening failure occurs. For example, a failure
   can be triggered when a connection to an external subtree source is
   broken.


.. _design-producer:

Data Tree Producer
------------------

A Data Tree Producer represents source of data in system. Data TreeProducer
implementations are not required to implement a specific interface, but
use a :mdsal-apidoc:`DOMDataTreeProducer <DOMDataTreeProducer.html>` instance
to publish data (i.e. to modify the Conceptual Data Tree).

A Data Tree Producer is exclusively bound to one or more subtrees of the
Conceptual Data Tree, i.e. binding a Data Tree Producer to a subtree prevents
other Data Tree Producers from modifying the subtree.

* A failed Data Tree Producer still holds a claim to the namespace to which
  it is bound (i.e. the exclusive lock of the subtree) until it is closed.

:mdsal-apidoc:`DOMDataTreeProducer <DOMDataTreeProducer.html>` represents a
Data Tree Producer context

* allows transactions to be submitted to subtrees specified at creation
  time
* at any given time there may be a single transaction open.
* once a transaction is submitted, it will proceed to be committed
  asynchronously.

.. todo:: Consider linking / inlining interface

.DOMDataTreeProducer interface signature

.. code-block:: java

   public interface DOMDataTreeProducer extends DOMDataTreeProducerFactory, AutoCloseable {
       DOMDataWriteTransaction createTransaction(boolean isolated); // (1)
       DOMDataTreeProducer createProducer(Collection<DOMDataTreeIdentifier> subtrees); // (2)
   }

#. Allocates a new transaction. All previously allocated transactions must
   have been either submitted or canceled. Setting `isolated` to `true`
   disables state compression for this transaction.
#. Creates a sub-producer for the provided `subtrees`. The parent producer
   loses the ability to access the specified paths until the resulting child
   producer is closed.

.. _design-shard:

Data Tree Shard
---------------

- **A Data Tree Shard** is always bound to either the ``OPERATIONAL``, or the
  ``CONFIG`` space, never to both at the same time.

- **Data Tree Shards** may be nested, the parent shard must be aware of
  sub-shards and execute every request in context of a self-consistent view of
  sub-shards liveness. Data requests passing through it must be multiplexed
  with sub-shard creations/deletions. In other words, if an application creates
  a transaction rooted at the parent Shard and attempts to access data residing
  in a sub-shard, the parent Shard implementation must coordinate with the
  sub-shard implementation to provide the illusion that the data resides in the
  parent shard. In the case of a transaction running concurrently with
  sub-shard creation or deletion, these operations need to execute atomically
  with respect to each other, which is to say that the transactions must
  completely execute as if the sub-shard creation/deletion occurred before the
  transaction started or as if the transaction completed before the sub-shard
  creation/deletion request was executed. This requirement can also be
  satisfied by the Shard implementation preventing transactions from
  completing. A Shard implementation may choose to abort any open transactions
  prior to executing a sub-shard operation.

- **Shard Layout** is local to an OpenDaylight instance.

- **Shard Layout** is modified by agents (registering / unregistering Data Tree
  Shards) in order to make the Data Tree Shard and the underlaying data
  available to plugins and applications executing on that particular
  OpenDaylight instance.

Registering a Shard with the Conceptual Data Tree
-------------------------------------------------

.. note:: Namespace in this context means a Data Tree Identifier prefix.

#. **Claim a namespace** - An agent that is registering a shard must prove that
   it has sufficient rights to modify the subtree where the shard is going to be
   attached. A namespace for the shard is claimed by binding a Data Tree
   Producer instance to same subtree where the shard will be bound. The Data
   Tree Producer must not have any open child producers, and it should not have
   any outstanding transactions.

#. **Create a shard instance** - Once a namespace is claimed, the agent creates
   a shard instance.

#. **Attach shard** - The agent registers the created shard instance and
   provides in the reigstration the Data Tree Producer instance to verify the
   namespace claim. The newly created Shard is checked for its ability to
   cooperate with its parent shard. If the check is successful, the newly
   created Shard is attached to its parent shard and recorded in the Shard
   layout.

#. **Remove namespace claim** (optional) - If the Shard is providing storage for
   applications, the agent should close the Data Tree Producer instance to make
   the subtree available to applications.

.. important::

   Steps 1, 2 and 3 may fail, and the recovery strategy depends
   on which step failed and on the failure reason.

.. todo:: Describe possible failures and recovery scenarios