= Conceptual Data Tree Robert Varga :rfc6020: https://tools.ietf.org/html/rfc6020 :mdsal-apidoc: https://nexus.opendaylight.org/content/sites/site/org.opendaylight.mdsal/boron/apidocs/org/opendaylight/mdsal/dom/api/ :toclevel: 3 :toc: == Terminology Data Tree:: The instatiated logical tree representing configuration or operational state data of modeled problem domain (e.g controller, network) Data Tree Consumer:: Component acting on data, after this data are introduced into particular subtree(s) of Data Tree. Data Tree Identifier:: A unique identifier for a particular subtree of Data Tree. It is composed of the logical data store type and the instance identifier of the root node. It is represented by `DOMDataTreeIdentifier`. Data Tree Producer:: Component responsible for providing data for particular subtree(s) of Data Tree. Data Tree Shard:: Component responsible for providing storage or access to particular subtree of Data Tree. Shard Layout:: Longest-prefix mapping between Data Tree Identifiers and Data Tree Shards responsible for providing access to data subtree. == Basic Concepts === Data Tree is a Namespace The concept of a data tree comes from {rfc6020}[RFC6020]. It is is vaguely split into two instances, configuration and operational. The implicit assumption is that *config implies oper*, e.g. any configuration data is also valid operational data. Further interactions between the two are left undefined and YANG language is not strictly extensible in the number and semantics of these instances, leaving a lot to implementation details. An outline of use, consistent with current MD-SAL design, is outlined in https://tools.ietf.org/html/draft-kwatsen-netmod-opstate[draft-kwatsen-netmod-opstate]. OpenDaylight MD-SAL design makes no inherent assumptions about config/oper relationship, treating them as separate entities and making them fully addressable (via `DOMDataTreeIdentifier`). It is up to MD-SAL plugins (e.g. protocol plugins, applications) to maintain this relationship. This reflects the asynchronous nature of applying configuration and also the fact that intended configuration data may be subject to translation (such as template configuration instantiation). Each of configuration and operational spaces are instances of the Conceptual Data Tree. Any data item in the conceptual data tree is addressed via `YangInstanceIdentifier`, which is a hierarchical. content-based, unique identifier. All applications identify data using these identifiers to MD-SAL services, which are expect to perform proper namespace management such that logical operation connectivity is maintained. === Identifiers versus Locators It is important to note that when we talk about Identifiers and Locators, we *do not* mean https://en.wikipedia.org/wiki/Uniform_Resource_Identifier[URIs and URLs], but rather URNs and URLs as strictly separate entities. MD-SAL plugins do not have access to locators and it is the job of MD-SAL services to provide location independence. Exact details on how a particular MD-SAL service achieves location independence is currently left up to the implementation, which leads to the problem of having MD-SAL service cooperate, such as storing data in different backends (in-memory, SQL, NoSQL, etc.) and providing unified access to all available data. Note that data availability is subject to capabilities of the storage engine and its operational state, which leads to the design decision that `YangInstanceIdentifier` needs to be performed in two steps: . a longest-prefix match is performed to locate the storage backend instance for that identifier . masked path elements are resolved by the storage backend. === Data Tree Shard A process similar to the first step is performed today by the Distributed Data Store implementation to split data into Shards. The concept of a shard as currently implemented is limited to specifying namespaces and does not allow pluggable storage engines. In the context of the conceptual data tree, the Shard concept is generalized as the shorthad equivalent of a storage backend instance. Shards can be attached at any (even wildcard) `YangInstanceIdentifier`. This contract is exposed via the `DOMShardedDataTree`, which is an MD-SAL SPI class implementing an `YangInstanceIdentifier` -> `Shard` registry service. This is an omnipresent MD-SAL service, Shard Registry, whose visibility scope is a single OpenDaylight instance (e.g. a cluster member). *Shard Layout* is used to refer to the mapping information contained in this service. === Federation, Replication and High Availability Support for various multi-node scenarios is a concern outside of core MD-SAL. If a particular scenario requires the shard layout to be replicated (either fully or partially), it is up to the Shard providers to maintain an omnipresent service on each node, which in turn is responsible for dynamically registering `DOMDataTreeShard` instances with Shard Registry. Since the Shard Layout is strictly local to a particular OpenDaylight instance, an OpenDaylight cluster is geared towards being asymmetric with each node serving its own view of the data. This allows each node to project its local operational details, as well as partitioning of the data set being worked on based on workload and node availability. Partial symmetry of the conceptual data tree can still be maintained to the extent that a particular deployment requires. == Design [[design-listener]] ==== Data Tree Listener Data Tree Listener is data consumer, e.g. process wanting to act on data after it has been introduced to the conceptual data tree. Data Tree Listener implements {mdsal-apidoc}DOMDataTreeListener.html[DOMDataTreeListener] interface and registers itself using {mdsal-apidoc}DOMDataTreeService.html[DOMDataTreeService]. Data Tree Listener may register for multiple subtrees and each time it is invoked it will be provided with current state of all subtrees, it is listening for. // FIXME: Consider linking / inlining interface .DOMDataTreeListener interface signature [source, java] ---- public interface DOMDataTreeListener extends EventListener { void onDataTreeChanged(Collection changes, // (1) Map> subtrees); void onDataTreeFailed(Collection causes); // (2) } ---- <1> Invoked when subscribed data tree changed. `changes` contains collection of changes, `subtrees` contains current state of all subtrees to which listener is registered. <2> Invoked when a subtree listening failure occurs. This can be triggered, for example, when a connection to external subtree source is broken. [[design-producer]] ==== Data Tree Producer Data Tree Producer represents source of data in system. Producer implementations are not required to implement specific interface, but uses a {mdsal-apidoc}DOMDataTreeProducer.html[DOMDataTreeProducer] instance to publish data (modify conceptual data tree). Data Tree Producer is exclusively bound to subtree(s) of Conceptual Data Tree, which prevents other producers to modify same subtree. * Failed Data Tree Producer still holds namespace claim (exclusive lock of subtree) till it is closed. {mdsal-apidoc}DOMDataTreeProducer.html[DOMDataTreeProducer] Represents Data producer context * allows transactions to be submitted to the subtrees specified at creation time * at any given time there may be a single transaction open. * once a transaction is submitted, it will proceed to be committed asynchronously. // FIXME: Consider linking / inlining interface .DOMDataTreeProducer interface signature [source, java] ---- public interface DOMDataTreeProducer extends DOMDataTreeProducerFactory, AutoCloseable { DOMDataWriteTransaction createTransaction(boolean isolated); // (1) DOMDataTreeProducer createProducer(Collection subtrees); // (2) } ---- <1> Allocates new transaction, all transactions previously allocated must have been either submitted or canceled. Setting `isolated` to `true` disables state compression for this transaction. <2> Creates sub-producer for provided `subtrees`, parent producer loses the ability to access the specified paths until the resulting (child) producer is shut down. [[design-shard]] === Data Tree Shard - *Data Tree Shard* is always bound to either `OPERATIONAL`, or `CONFIG` space, never combined. - *Data Tree Shards* may be nested, parent shard must be aware of sub-shards and has to execute every request in the context of a self-consistent view of sub-shards liveness. Data requests passing through it need to be multiplexed with sub-shard creation/deletion. - *Shard Layout* is local to OpenDaylight instance. - *Shard Layout* is modified by agents (registering / unregistering Data Tree Shards) in order to make Data Tree Shard and underlaying data available to local instance ==== Registering Shard NOTE: Namespace in this context means Data Tree Identifier prefix . *Claim namespace* - Agent registering shard must prove that it has sufficient rights to modify subtree where shard is going to be attached. Namespace claim is realized by having Data Tree Producer instance which is bound to same subtree as shard, and producer must not have any open child producers and should not have any outstanding transactions. . *Create shard instance* - Once namespace is claimed, agent creates shard instance. . *Attach shard* - Agent registers created shard instance along with providing Data Tree Producer instance to verify namespace claim. Shard is verified if it is able to cooperate with parent shard and then it is attached to parent shard and Shard layout. . *Remove namespace claim* (optional) - If shard is providing storage for applications, agent should close Data Tree Producer instance in order to make subtree available to applications. IMPORTANT: Step 1, 2 and 3 may fail and recovery strategy depends on step which failed and failure reason. // FIXME: Describe possible failures and recovery scenarios