docs/developer-guide/topology-processing-framework-developer-guide.rst

   1 .. _topoprocessing-dev-guide:
   2
   3 Topology Processing Framework Developer Guide
   4 =============================================
   5
   6 Overview
   7 --------
   8
   9 The Topology Processing Framework allows developers to aggregate and
  10 filter topologies according to defined correlations. It also provides
  11 functionality, which you can use to make your own topology model by
  12 automating the translation from one model to another. For example to
  13 translate from the opendaylight-inventory model to only using the
  14 network-topology model.
  15
  16 Architecture
  17 ------------
  18
  19 Chapter Overview
  20 ~~~~~~~~~~~~~~~~
  21
  22 In this chapter we describe the architecture of the Topology Processing
  23 Framework. In the first part, we provide information about available
  24 features and basic class relationships. In the second part, we describe
  25 our model specific approach, which is used to provide support for
  26 different models.
  27
  28 Basic Architecture
  29 ~~~~~~~~~~~~~~~~~~
  30
  31 The Topology Processing Framework consists of several Karaf features:
  32
  33 -  odl-topoprocessing-framework
  34
  35 -  odl-topoprocessing-inventory
  36
  37 -  odl-topoprocessing-network-topology
  38
  39 -  odl-topoprocessing-i2rs
  40
  41 -  odl-topoprocessing-inventory-rendering
  42
  43 The feature odl-topoprocessing-framework contains the
  44 topoprocessing-api, topoprocessing-spi and topoprocessing-impl bundles.
  45 This feature is the core of the Topology Processing Framework and is
  46 required by all others features.
  47
  48 -  topoprocessing-api - contains correlation definitions and definitions
  49    required for rendering
  50
  51 -  topoprocessing-spi - entry point for topoprocessing service (start
  52    and close)
  53
  54 -  topoprocessing-impl - contains base implementations of handlers,
  55    listeners, aggregators and filtrators
  56
  57 TopoProcessingProvider is the entry point for Topology Processing
  58 Framework. It requires a DataBroker instance. The DataBroker is needed
  59 for listener registration. There is also the TopologyRequestListener
  60 which listens on aggregated topology requests (placed into the
  61 configuration datastore) and UnderlayTopologyListeners which listen on
  62 underlay topology data changes (made in operational datastore). The
  63 TopologyRequestHandler saves toporequest data and provides a method for
  64 translating a path to the specified leaf. When a change in the topology
  65 occurs, the registered UnderlayTopologyListener processes this
  66 information for further aggregation and/or filtration. Finally, after an
  67 overlay topology is created, it is passed to the TopologyWriter, which
  68 writes this topology into operational datastore.
  69
  70 .. figure:: ./images/topoprocessing/TopologyRequestHandler_classesRelationship.png
  71    :alt: Class relationship
  72
  73    Class relationship
  74
  75 [1] TopologyRequestHandler instantiates TopologyWriter and
  76 TopologyManager. Then, according to the request, initializes either
  77 TopologyAggregator, TopologyFiltrator or LinkCalculator.
  78
  79 [2] It creates as many instances of UnderlayTopologyListener as there
  80 are underlay topologies.
  81
  82 [3] PhysicalNodes are created for relevant incoming nodes (those having
  83 node ID).
  84
  85 [4a] It performs aggregation and creates logical nodes.
  86
  87 [4b] It performs filtration and creates logical nodes.
  88
  89 [4c] It performs link computation and creates links between logical
  90 nodes.
  91
  92 [5] Logical nodes are put into wrapper.
  93
  94 [6] The wrapper is translated into the appropriate format and written
  95 into datastore.
  96
  97 Model Specific Approach
  98 ~~~~~~~~~~~~~~~~~~~~~~~
  99
 100 The Topology Processing Framework consists of several modules and Karaf
 101 features, which provide support for different input models. Currently we
 102 support the network-topology, opendaylight-inventory and i2rs models.
 103 For each of these input models, the Topology Processing Framework has
 104 one module and one Karaf feature.
 105
 106 How it works
 107 ^^^^^^^^^^^^
 108
 109 **User point of view:**
 110
 111 When you start the odl-topoprocessing-framework feature, the Topology
 112 Processing Framework starts without knowledge how to work with any input
 113 models. In order to allow the Topology Processing Framework to process
 114 some kind of input model, you must install one (or more) model specific
 115 features. Installing these features will also start
 116 odl-topoprocessing-framework feature if it is not already running. These
 117 features inject appropriate logic into the odl-topoprocessing-framework
 118 feature. From that point, the Topology Processing Framework is able to
 119 process different kinds of input models, specifically those that you
 120 install features for.
 121
 122 **Developer point of view:**
 123
 124 The topoprocessing-impl module contains (among other things) classes and
 125 interfaces, which are common for every model specific topoprocessing
 126 module. These classes and interfaces are implemented and extended by
 127 classes in particular model specific modules. Model specific modules
 128 also depend on the TopoProcessingProvider class in the
 129 topoprocessing-spi module. This dependency is injected during
 130 installation of model specific features in Karaf. When a model specific
 131 feature is started, it calls the registerAdapters(adapters) method of
 132 the injected TopoProcessingProvider object. After this step, the
 133 Topology Processing Framework is able to use registered model adapters
 134 to work with input models.
 135
 136 To achieve the described functionality we created a ModelAdapter
 137 interface. It represents installed feature and provides methods for
 138 creating crucial structures specific to each model.
 139
 140 .. figure:: ./images/topoprocessing/ModelAdapter.png
 141    :alt: ModelAdapter interface
 142
 143    ModelAdapter interface
 144
 145 Model Specific Features
 146 ^^^^^^^^^^^^^^^^^^^^^^^
 147
 148 -  odl-topoprocessing-network-topology - this feature contains logic to
 149    work with network-topology model
 150
 151 -  odl-topoprocessing-inventory - this feature contains logic to work
 152    with opendaylight-inventory model
 153
 154 -  odl-topoprocessing-i2rs - this feature contains logic to work with
 155    i2rs model
 156
 157 Inventory Model Support
 158 ~~~~~~~~~~~~~~~~~~~~~~~
 159
 160 The opendaylight-inventory model contains only nodes, termination
 161 points, information regarding these structures. This model co-operates
 162 with network-topology model, where other topology related information is
 163 stored. This means that we have to handle two input models at once. To
 164 support the inventory model, InventoryListener and
 165 NotificationInterConnector classes were introduced. Please see the flow
 166 diagrams below.
 167
 168 .. figure:: ./images/topoprocessing/Network_topology_model_flow_diagram.png
 169    :alt: Network topology model
 170
 171    Network topology model
 172
 173 .. figure:: ./images/topoprocessing/Inventory_model_listener_diagram.png
 174    :alt: Inventory model
 175
 176    Inventory model
 177
 178 Here we can see the InventoryListener and NotificationInterConnector
 179 classes. InventoryListener listens on data changes in the inventory
 180 model and passes these changes wrapped as an UnderlayItem for further
 181 processing to NotificationInterConnector. It doesn’t contain node
 182 information - it contains a leafNode (node based on which aggregation
 183 occurs) instead. The node information is stored in the topology model,
 184 where UnderlayTopologyListener is registered as usual. This listener
 185 delivers the missing information.
 186
 187 Then the NotificationInterConnector combines the two notifications into
 188 a complete UnderlayItem (no null values) and delivers this UnderlayItem
 189 for further processing (to next TopologyOperator).
 190
 191 Aggregation and Filtration
 192 --------------------------
 193
 194 Chapter Overview
 195 ~~~~~~~~~~~~~~~~
 196
 197 The Topology Processing Framework allows the creation of aggregated
 198 topologies and filtered views over existing topologies. Currently,
 199 aggregation and filtration is supported for topologies that follow
 200 `network-topology <https://github.com/opendaylight/yangtools/blob/master/model/ietf/ietf-topology/src/main/yang/network-topology%402013-10-21.yang>`__,
 201 opendaylight-inventory or i2rs model. When a request to create an
 202 aggregated or filtered topology is received, the framework creates one
 203 listener per underlay topology. Whenever any specified underlay topology
 204 is changed, the appropriate listener is triggered with the change and
 205 the change is processed. Two types of correlations (functionalities) are
 206 currently supported:
 207
 208 -  Aggregation
 209
 210    -  Unification
 211
 212    -  Equality
 213
 214 -  Filtration
 215
 216 Terminology
 217 ~~~~~~~~~~~
 218
 219 We use the term underlay item (physical node) for items (nodes, links,
 220 termination-points) from underlay and overlay item (logical node) for
 221 items from overlay topologies regardless of whether those are actually
 222 physical network elements.
 223
 224 Aggregation
 225 ~~~~~~~~~~~
 226
 227 Aggregation is an operation which creates an aggregated item from two or
 228 more items in the underlay topology if the aggregation condition is
 229 fulfilled. Requests for aggregated topologies must specify a list of
 230 underlay topologies over which the overlay (aggregated) topology will be
 231 created and a target field in the underlay item that the framework will
 232 check for equality.
 233
 234 Create Overlay Node
 235 ^^^^^^^^^^^^^^^^^^^
 236
 237 First, each new underlay item is inserted into the proper topology
 238 store. Once the item is stored, the framework compares it (using the
 239 target field value) with all stored underlay items from underlay
 240 topologies. If there is a target-field match, a new overlay item is
 241 created containing pointers to all *equal* underlay items. The newly
 242 created overlay item is also given new references to its supporting
 243 underlay items.
 244
 245 **Equality case:**
 246
 247 If an item doesn’t fulfill the equality condition with any other items,
 248 processing finishes after adding the item into topology store. It will
 249 stay there for future use, ready to create an aggregated item with a new
 250 underlay item, with which it would satisfy the equality condition.
 251
 252 **Unification case:**
 253
 254 An overlay item is created for all underlay items, even those which
 255 don’t fulfill the equality condition with any other items. This means
 256 that an overlay item is created for every underlay item, but for items
 257 which satisfy the equality condition, an aggregated item is created.
 258
 259 Update Node
 260 ^^^^^^^^^^^
 261
 262 Processing of updated underlay items depends on whether the target field
 263 has been modified. If yes, then:
 264
 265 -  if the underlay item belonged to some overlay item, it is removed
 266    from that item. Next, if the aggregation condition on the target
 267    field is satisfied, the item is inserted into another overlay item.
 268    If the condition isn’t met then:
 269
 270    -  in equality case - the item will not be present in overlay
 271       topology.
 272
 273    -  in unification case - the item will create an overlay item with a
 274       single underlay item and this will be written into overlay
 275       topology.
 276
 277 -  if the item didn’t belong to some overlay item, it is checked again
 278    for aggregation with other underlay items.
 279
 280 Remove Node
 281 ^^^^^^^^^^^
 282
 283 The underlay item is removed from the corresponding topology store, from
 284 it’s overlay item (if it belongs to one) and this way it is also removed
 285 from overlay topology.
 286
 287 **Equality case:**
 288
 289 If there is only one underlay item left in the overlay item, the overlay
 290 item is removed.
 291
 292 **Unification case:**
 293
 294 The overlay item is removed once it refers to no underlay item.
 295
 296 Filtration
 297 ~~~~~~~~~~
 298
 299 Filtration is an operation which results in creation of overlay topology
 300 containing only items fulfilling conditions set in the topoprocessing
 301 request.
 302
 303 Create Underlay Item
 304 ^^^^^^^^^^^^^^^^^^^^
 305
 306 If a newly created underlay item passes all filtrators and their
 307 conditions, then it is stored in topology store and a creation
 308 notification is delivered into topology manager. No operation otherwise.
 309
 310 Update Underlay Item
 311 ^^^^^^^^^^^^^^^^^^^^
 312
 313 First, the updated item is checked for presence in topology store:
 314
 315 -  if it is present in topology store:
 316
 317    -  if it meets the filtering conditions, then processUpdatedData
 318       notification is triggered
 319
 320    -  else processRemovedData notification is triggered
 321
 322 -  if item isn’t present in topology store
 323
 324    -  if item meets filtering conditions, then processCreatedData
 325       notification is triggered
 326
 327    -  else it is ignored
 328
 329 Remove Underlay Item
 330 ^^^^^^^^^^^^^^^^^^^^
 331
 332 If an underlay node is supporting some overlay node, the overlay node is
 333 simply removed.
 334
 335 Default Filtrator Types
 336 ^^^^^^^^^^^^^^^^^^^^^^^
 337
 338 There are seven types of default filtrators defined in the framework:
 339
 340 -  IPv4-address filtrator - checks if specified field meets IPv4 address
 341    + mask criteria
 342
 343 -  IPv6-address filtrator - checks if specified field meets IPv6 address
 344    + mask criteria
 345
 346 -  Specific number filtrator - checks for specific number
 347
 348 -  Specific string filtrator - checks for specific string
 349
 350 -  Range number filtrator - checks if specified field is higher than
 351    provided minimum (inclusive) and lower than provided maximum
 352    (inclusive)
 353
 354 -  Range string filtrator - checks if specified field is alphabetically
 355    greater than provided minimum (inclusive) and alphabetically lower
 356    than provided maximum (inclusive)
 357
 358 -  Script filtrator - allows a user or application to implement their
 359    own filtrator
 360
 361 Register Custom Filtrator
 362 ^^^^^^^^^^^^^^^^^^^^^^^^^
 363
 364 There might be some use case that cannot be achieved with the default
 365 filtrators. In these cases, the framework offers the possibility for a
 366 user or application to register a custom filtrator.
 367
 368 Pre-Filtration / Filtration & Aggregation
 369 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 370
 371 This feature was introduced in order to lower memory and performance
 372 demands. It is a combination of the filtration and aggregation
 373 operations. First, uninteresting items are filtered out and then
 374 aggregation is performed only on items that passed filtration. This way
 375 the framework saves on compute time. The PreAggregationFiltrator and
 376 TopologyAggregator share the same TopoStoreProvider (and thus topology
 377 store) which results in lower memory demands (as underlay items are
 378 stored only in one topology store - they aren’t stored twice).
 379
 380 Link Computation
 381 ----------------
 382
 383 Chapter Overview
 384 ~~~~~~~~~~~~~~~~
 385
 386 While processing the topology request, we create overlay nodes with
 387 lists of supporting underlay nodes. Because these overlay nodes have
 388 completely new identifiers, we lose link information. To regain this
 389 link information, we provide Link Computation functionality. Its main
 390 purpose is to create new overlay links based on the links from the
 391 underlay topologies and underlay items from overlay items. The required
 392 information for Link Computation is provided via the Link Computation
 393 model in
 394 (`topology-link-computation.yang <https://git.opendaylight.org/gerrit/gitweb?p=topoprocessing.git;a=blob;f=topoprocessing-api/src/main/yang/topology-link-computation.yang;hb=refs/heads/stable/boron>`__).
 395
 396 Link Computation Functionality
 397 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 398
 399 Let us consider two topologies with following components:
 400
 401 Topology 1:
 402
 403 -  Node: ``node:1:1``
 404
 405 -  Node: ``node:1:2``
 406
 407 -  Node: ``node:1:3``
 408
 409 -  Link: ``link:1:1`` (from ``node:1:1`` to ``node:1:2``)
 410
 411 -  Link: ``link:1:2`` (from ``node:1:3`` to ``node:1:2``)
 412
 413 Topology 2:
 414
 415 -  Node: ``node:2:1``
 416
 417 -  Node: ``node:2:2``
 418
 419 -  Node: ``node:2:3``
 420
 421 -  Link: ``link:2:1`` (from ``node:2:1`` to ``node:2:3``)
 422
 423 Now let’s say that we applied some operations over these topologies that
 424 results into aggregating together
 425
 426 -  ``node:1:1`` and ``node:2:3`` (``node:1``)
 427
 428 -  ``node:1:2`` and ``node:2:2`` (``node:2``)
 429
 430 -  ``node:1:3`` and ``node:2:1`` (``node:3``)
 431
 432 At this point we can no longer use available links in new topology
 433 because of the node ID change, so we must create new overlay links with
 434 source and destination node set to new nodes IDs. It means that
 435 ``link:1:1`` from topology 1 will create new link ``link:1``. Since
 436 original source (``node:1:1``) is already aggregated under ``node:1``,
 437 it will become source node for ``link:1``. Using same method the
 438 destination will be ``node:2``. And the final output will be three
 439 links:
 440
 441 -  ``link:1``, from ``node:1`` to ``node:2``
 442
 443 -  ``link:2``, from ``node:3`` to ``node:2``
 444
 445 -  ``link:3``, from ``node:3`` to ``node:1``
 446
 447 .. figure:: ./images/topoprocessing/LinkComputation.png
 448    :alt: Overlay topology with computed links
 449
 450    Overlay topology with computed links
 451
 452 In-Depth Look
 453 ~~~~~~~~~~~~~
 454
 455 The main logic behind Link Computation is executed in the LinkCalculator
 456 operator. The required information is passed to LinkCalculator through
 457 the LinkComputation section of the topology request. This section is
 458 defined in the topology-link-computation.yang file. The main logic also
 459 covers cases when some underlay nodes may not pass through other
 460 topology operators.
 461
 462 Link Computation Model
 463 ^^^^^^^^^^^^^^^^^^^^^^
 464
 465 There are three essential pieces of information for link computations.
 466 All of them are provided within the LinkComputation section. These
 467 pieces are:
 468
 469 -  output model
 470
 471 .. code:: yang
 472
 473     leaf output-model {
 474         type identityref {
 475             base topo-corr:model;
 476         }
 477         description "Desired output model for computed links.";
 478     }
 479
 480 -  overlay topology with new nodes
 481
 482 .. code:: yang
 483
 484     container node-info {
 485         leaf node-topology {
 486             type string;
 487             mandatory true;
 488             description "Topology that contains aggregated nodes.
 489                          This topology will be used for storing computed links.";
 490         }
 491         uses topo-corr:input-model-grouping;
 492     }
 493
 494 -  underlay topologies with original links
 495
 496 .. code:: yang
 497
 498     list link-info {
 499         key "link-topology input-model";
 500         leaf link-topology {
 501             type string;
 502             mandatory true;
 503             description "Topology that contains underlay (base) links.";
 504         }
 505         leaf aggregated-links {
 506             type boolean;
 507             description "Defines if link computation should be based on supporting-links.";
 508         }
 509         uses topo-corr:input-model-grouping;
 510     }
 511
 512 This whole section is augmented into ``network-topology:topology``. By
 513 placing this section out of correlations section, it allows us to send
 514 link computation request separately from topology operations request.
 515
 516 Main Logic
 517 ^^^^^^^^^^
 518
 519 Taking into consideration that some of the underlay nodes may not
 520 transform into overlay nodes (e.g. they are filtered out), we created
 521 two possible states for links:
 522
 523 -  matched - a link is considered as matched when both original source
 524    and destination node were transformed to overlay nodes
 525
 526 -  waiting - a link is considered as waiting if original source,
 527    destination or both nodes are missing from the overlay topology
 528
 529 All links in waiting the state are stored in waitingLinks list, already
 530 matched links are stored in matchedLinks list and overlay nodes are
 531 stored in the storedOverlayNodes list. All processing is based only on
 532 information in these lists. Processing created, updated and removed
 533 underlay items is slightly different and described in next sections
 534 separately.
 535
 536 **Processing Created Items**
 537
 538 Created items can be either nodes or links, depending on the type of
 539 listener from which they came. In the case of a link, it is immediately
 540 added to waitingLinks and calculation for possible overlay link
 541 creations (calculatePossibleLink) is started. The flow diagram for this
 542 process is shown in the following picture:
 543
 544 .. figure:: ./images/topoprocessing/LinkComputationFlowDiagram.png
 545    :alt: Flow diagram of processing created items
 546
 547    Flow diagram of processing created items
 548
 549 Searching for the source and destination nodes in the
 550 calculatePossibleLink method runs over each node in storedOverlayNodes
 551 and the IDs of each supporting node is compared against IDs from the
 552 underlay link’s source and destination nodes. If there are any nodes
 553 missing, the link remains in the waiting state. If both the source and
 554 destination nodes are found, the corresponding overlay nodes is recorded
 555 as the new source and destination. The link is then removed from
 556 waitingLinks and a new CalculatedLink is added to the matched links. At
 557 the end, the new link (if it exists) is written into the datastore.
 558
 559 If the created item is an overlayNode, this is added to
 560 storedOverlayNodes and we call calculatePossibleLink for every link in
 561 waitingLinks.
 562
 563 **Processing Updated Items**
 564
 565 The difference from processing created items is that we have three
 566 possible types of updated items: overlay nodes, waiting underlay links,
 567 and matched underlay links.
 568
 569 -  In the case of a change in a matched link, this must be recalculated
 570    and based on the result it will either be matched with new source and
 571    destination or will be returned to waiting links. If the link is
 572    moved back to a waiting state, it must also be removed from the
 573    datastore.
 574
 575 -  In the case of change in a waiting link, it is passed to the
 576    calculation process and based on the result will either remain in
 577    waiting state or be promoted to the matched state.
 578
 579 -  In the case of a change in an overlay node, storedOverlayNodes must
 580    be updated properly and all links must be recalculated in case of
 581    changes.
 582
 583 **Processing Removed items**
 584
 585 Same as for processing updated item. There can be three types of removed
 586 items:
 587
 588 -  In case of waiting link removal, the link is just removed from
 589    waitingLinks
 590
 591 -  In case of matched link removal, the link is removed from
 592    matchingLinks and datastore
 593
 594 -  In case of overlay node removal, the node must be removed form
 595    storedOverlayNodes and all matching links must be recalculated
 596
 597 Wrapper, RPC Republishing, Writing Mechanism
 598 --------------------------------------------
 599
 600 Chapter Overview
 601 ~~~~~~~~~~~~~~~~
 602
 603 During the process of aggregation and filtration, overlay items (so
 604 called logical nodes) were created from underlay items (physical nodes).
 605 In the topology manager, overlay items are put into a wrapper. A wrapper
 606 is identified with unique ID and contains list of logical nodes.
 607 Wrappers are used to deal with transitivity of underlay items - which
 608 permits grouping of overlay items (into wrappers).
 609
 610 .. figure:: ./images/topoprocessing/wrapper.png
 611    :alt: Wrapper
 612
 613    Wrapper
 614
 615 PN1, PN2, PN3 = physical nodes
 616
 617 LN1, LN2 = logical nodes
 618
 619 RPC Republishing
 620 ~~~~~~~~~~~~~~~~
 621
 622 All RPCs registered to handle underlay items are re-registered under
 623 their corresponding wrapper ID. RPCs of underlay items (belonging to an
 624 overlay item) are gathered, and registered under ID of their wrapper.
 625
 626 RPC Call
 627 ^^^^^^^^
 628
 629 When RPC is called on overlay item, this call is delegated to it’s
 630 underlay items, this means that the RPC is called on all underlay items
 631 of this overlay item.
 632
 633 Writing Mechanism
 634 ~~~~~~~~~~~~~~~~~
 635
 636 When a wrapper (containing overlay item(s) with it’s underlay item(s))
 637 is ready to be written into data store, it has to be converted into DOM
 638 format. After this translation is done, the result is written into
 639 datastore. Physical nodes are stored as supporting-nodes. In order to
 640 use resources responsibly, writing operation is divided into two steps.
 641 First, a set of threads registers prepared operations (deletes and puts)
 642 and one thread makes actual write operation in batch.
 643
 644 Topology Rendering Guide - Inventory Rendering
 645 ----------------------------------------------
 646
 647 Chapter Overview
 648 ~~~~~~~~~~~~~~~~
 649
 650 In the most recent OpenDaylight release, the opendaylight-inventory
 651 model is marked as deprecated. To facilitate migration from it to the
 652 network-topology model, there were requests to render (translate) data
 653 from inventory model (whether augmented or not) to another model for
 654 further processing. The Topology Processing Framework was extended to
 655 provide this functionality by implementing several rendering-specific
 656 classes. This chapter is a step-by-step guide on how to implement your
 657 own topology rendering using our inventory rendering as an example.
 658
 659 Use case
 660 ~~~~~~~~
 661
 662 For the purpose of this guide we are going to render the following
 663 augmented fields from the OpenFlow model:
 664
 665 -  from inventory node:
 666
 667    -  manufacturer
 668
 669    -  hardware
 670
 671    -  software
 672
 673    -  serial-number
 674
 675    -  description
 676
 677    -  ip-address
 678
 679 -  from inventory node-connector:
 680
 681    -  name
 682
 683    -  hardware-address
 684
 685    -  current-speed
 686
 687    -  maximum-speed
 688
 689 We also want to preserve the node ID and termination-point ID from
 690 opendaylight-topology-inventory model, which is network-topology part of
 691 the inventory model.
 692
 693 Implementation
 694 ~~~~~~~~~~~~~~
 695
 696 There are two ways to implement support for your specific topology
 697 rendering:
 698
 699 -  add a module to your project that depends on the Topology Processing
 700    Framework
 701
 702 -  add a module to the Topology Processing Framework itself
 703
 704 Regardless, a successful implementation must complete all of the
 705 following steps.
 706
 707 Step1 - Target Model Creation
 708 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 709
 710 Because the network-topology node does not have fields to store all
 711 desired data, it is necessary to create new model to render this extra
 712 data in to. For this guide we created the inventory-rendering model. The
 713 picture below shows how data will be rendered and stored.
 714
 715 .. figure:: ./images/topoprocessing/Inventory_Rendering_Use_case.png
 716    :alt: Rendering to the inventory-rendering model
 717
 718    Rendering to the inventory-rendering model
 719
 720 .. important::
 721
 722     When implementing your version of the topology-rendering model in
 723     the Topology Processing Framework, the source file of the model
 724     (.yang) must be saved in /topoprocessing-api/src/main/yang folder so
 725     corresponding structures can be generated during build and can be
 726     accessed from every module through dependencies.
 727
 728 When the target model is created you have to add an identifier through
 729 which you can set your new model as output model. To do that you have to
 730 add another identity item to topology-correlation.yang file. For our
 731 inventory-rendering model identity looks like this:
 732
 733 .. code:: yang
 734
 735     identity inventory-rendering-model {
 736         description "inventory-rendering.yang";
 737         base model;
 738     }
 739
 740 After that you will be able to set inventory-rendering-model as output
 741 model in XML.
 742
 743 Step2 - Module and Feature Creation
 744 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 745
 746 .. important::
 747
 748     This and following steps are based on the `model specific
 749     approach <#_model_specific_approach>`__ in the Topology Processing
 750     Framework. We highly recommend that you familiarize yourself with
 751     this approach in advance.
 752
 753 To create a base module and add it as a feature to Karaf in the Topology
 754 Processing Framework we made the changes in following
 755 `commit <https://git.opendaylight.org/gerrit/#/c/26223/>`__. Changes in
 756 other projects will likely be similar.
 757
 758 +--------------------------------------+--------------------------------------+
 759 | File                                 | Changes                              |
 760 +======================================+======================================+
 761 | pom.xml                              | add new module to topoprocessing     |
 762 +--------------------------------------+--------------------------------------+
 763 | features.xml                         | add feature to topoprocessing        |
 764 +--------------------------------------+--------------------------------------+
 765 | features/pom.xml                     | add dependencies needed by features  |
 766 +--------------------------------------+--------------------------------------+
 767 | topoprocessing-artifacts/pom.xml     | add artifact                         |
 768 +--------------------------------------+--------------------------------------+
 769 | topoprocessing-config/pom.xml        | add configuration file               |
 770 +--------------------------------------+--------------------------------------+
 771 | 81-topoprocessing-inventory-renderin | configuration file for new module    |
 772 | g-config.xml                         |                                      |
 773 +--------------------------------------+--------------------------------------+
 774 | topoprocessing-inventory-rendering/p | main pom for new module              |
 775 | om.xml                               |                                      |
 776 +--------------------------------------+--------------------------------------+
 777 | TopoProcessingProviderIR.java        | contains startup method which        |
 778 |                                      | register new model adapter           |
 779 +--------------------------------------+--------------------------------------+
 780 | TopoProcessingProviderIRModule.java  | generated class which contains       |
 781 |                                      | createInstance method. You should    |
 782 |                                      | call your startup method from here.  |
 783 +--------------------------------------+--------------------------------------+
 784 | TopoProcessingProviderIRModuleFactor | generated class. You will probably   |
 785 | y.java                               | not need to edit this file           |
 786 +--------------------------------------+--------------------------------------+
 787 | log4j.xml                            | configuration file for logger        |
 788 |                                      | topoprocessing-inventory-rendering-p |
 789 |                                      | rovider-impl.yang                    |
 790 +--------------------------------------+--------------------------------------+
 791
 792 Step3 - Module Adapters Creation
 793 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 794
 795 There are seven mandatory interfaces or abstract classes that needs to
 796 be implemented in each module. They are:
 797
 798 -  TopoProcessingProvider - provides module registration
 799
 800 -  ModelAdapter - provides model specific instances
 801
 802 -  TopologyRequestListener - listens on changes in the configuration
 803    datastore
 804
 805 -  TopologyRequestHandler - processes configuration datastore changes
 806
 807 -  UnderlayTopologyListener - listens for changes in the specific model
 808
 809 -  LinkTransaltor and NodeTranslator - used by OverlayItemTranslator to
 810    create NormalizedNodes from OverlayItems
 811
 812 The name convention we used was to add an abbreviation for the specific
 813 model to the beginning of implementing class name (e.g. the
 814 IRModelAdapter refers to class which implements ModelAdapter in module
 815 Inventory Rendering). In the case of the provider class, we put the
 816 abbreviation at the end.
 817
 818 .. important::
 819
 820     -  In the next sections, we use the terms TopologyRequestListener,
 821        TopologyRequestHandler, etc. without a prepended or appended
 822        abbreviation because the steps apply regardless of which specific
 823        model you are targeting.
 824
 825     -  If you want to implement rendering from inventory to
 826        network-topology, you can just copy-paste our module and
 827        additional changes will be required only in the output part.
 828
 829 **Provider part**
 830
 831 This part is the starting point of the whole module. It is responsible
 832 for creating and registering TopologyRequestListeners. It is necessary
 833 to create three classes which will import:
 834
 835 -  **TopoProcessingProviderModule** - is a generated class from
 836    topoprocessing-inventory-rendering-provider-impl.yang (created in
 837    previous step, file will appear after first build). Its method
 838    ``createInstance()`` is called at the feature start and must be
 839    modified to create an instance of TopoProcessingProvider and call its
 840    ``startup(TopoProcessingProvider topoProvider)`` function.
 841
 842 -  **TopoProcessingProvider** - in
 843    ``startup(TopoProcessingProvider topoProvider)`` function provides
 844    ModelAdapter registration to TopoProcessingProviderImpl.
 845
 846 -  **ModelAdapter** - provides creation of corresponding module specific
 847    classes.
 848
 849 **Input part**
 850
 851 This includes the creation of the classes responsible for input data
 852 processing. In this case, we had to create five classes implementing:
 853
 854 -  **TopologyRequestListener** and **TopologyRequestHandler** - when
 855    notified about a change in the configuration datastore, verify if the
 856    change contains a topology request (has correlations in it) and
 857    creates UnderlayTopologyListeners if needed. The implementation of
 858    these classes will differ according to the model in which are
 859    correlations saved (network-topology or i2rs). In the case of using
 860    network-topology, as the input model, you can use our classes
 861    IRTopologyRequestListener and IRTopologyRequestHandler.
 862
 863 -  **UnderlayTopologyListener** - registers underlay listeners according
 864    to input model. In our case (listening in the inventory model), we
 865    created listeners for the network-topology model and inventory model,
 866    and set the NotificationInterConnector as the first operator and set
 867    the IRRenderingOperator as the second operator (after
 868    NotificationInterConnector). Same as for
 869    TopologyRequestListener/Handler, if you are rendering from the
 870    inventory model, you can use our class IRUnderlayTopologyListener.
 871
 872 -  **InventoryListener** - a new implementation of this class is
 873    required only for inventory input model. This is because the
 874    InventoryListener from topoprocessing-impl requires pathIdentifier
 875    which is absent in the case of rendering.
 876
 877 -  **TopologyOperator** - replaces classic topoprocessing operator.
 878    While the classic operator provides specific operations on topology,
 879    the rendering operator just wraps each received UnderlayItem to
 880    OverlayItem and sends them to write.
 881
 882 .. important::
 883
 884     For purposes of topology rendering from inventory to
 885     network-topology, there are misused fields in UnderlayItem as
 886     follows:
 887
 888     -  item - contains node from network-topology part of inventory
 889
 890     -  leafItem - contains node from inventory
 891
 892     In case of implementing UnderlayTopologyListener or
 893     InventoryListener you have to carefully adjust UnderlayItem creation
 894     to these terms.
 895
 896 **Output part**
 897
 898 The output part of topology rendering is responsible for translating
 899 received overlay items to normalized nodes. In the case of inventory
 900 rendering, this is where node information from inventory are combined
 901 with node information from network-topology. This combined information
 902 is stored in our inventory-rendering model normalized node and passed to
 903 the writer.
 904
 905 The output part consists of two translators implementing the
 906 NodeTranslator and LinkTranslator interfaces.
 907
 908 **NodeTranslator implementation** - The NodeTranslator interface has one
 909 ``translate(OverlayItemWrapper wrapper)`` method. For our purposes,
 910 there is one important thing in wrapper - the list of OverlayItems which
 911 have one or more common UnderlayItems. Regardless of this list, in the
 912 case of rendering it will always contains only one OverlayItem. This
 913 item has list of UnderlayItems, but again in case of rendering there
 914 will be only one UnderlayItem item in this list. In NodeTranslator, the
 915 OverlayItem and corresponding UnderlayItem represent nodes from the
 916 translating model.
 917
 918 The UnderlayItem has several attributes. How you will use these
 919 attributes in your rendering is up to you, as you create this item in
 920 your topology operator. For example, as mentioned above, in our
 921 inventory rendering example is an inventory node normalized node stored
 922 in the UnderlayItem leafNode attribute, and we also store node-id from
 923 network-topology model in UnderlayItem itemId attribute. You can now use
 924 these attributes to build a normalized node for your new model. How to
 925 read and create normalized nodes is out of scope of this document.
 926
 927 **LinkTranslator implementation** - The LinkTranslator interface also
 928 has one ``translate(OverlayItemWrapper wrapper)`` method. In our
 929 inventory rendering this method returns ``null``, because the inventory
 930 model doesn’t have links. But if you also need links, this is the place
 931 where you should translate it into a normalized node for your model. In
 932 LinkTranslator, the OverlayItem and corresponding UnderlayItem represent
 933 links from the translating model. As in NodeTranslator, there will be
 934 only one OverlayItem and one UnderlayItem in the corresponding lists.
 935
 936 Testing
 937 ~~~~~~~
 938
 939 If you want to test topoprocessing with some manually created underlay
 940 topologies (like in this guide), than you have to tell Topoprocessing
 941 to listen for underlay topologies on Configuration datastore
 942 instead of Operational.
 943
 944 | You can do this in this config file
 945 | ``<topoprocessing_directory>/topoprocessing-config/src/main/resources/80-topoprocessing-config.xml``.
 946 | Here you have to change
 947 | ``<datastore-type>OPERATIONAL</datastore-type>``
 948 | to
 949 | ``<datastore-type>CONFIGURATION</datastore-type>``.
 950
 951
 952 Also you have to add dependency required to test "inventory" topologies.
 953
 954 | In ``<topoprocessing_directory>/features/pom.xml``
 955 | add ``<openflowplugin.version>latest_snapshot</openflowplugin.version>``
 956   to properties section
 957 | and add this dependency to dependencies section
 958
 959 .. code:: xml
 960
 961         <dependency>
 962                 <groupId>org.opendaylight.openflowplugin</groupId>
 963                 <artifactId>features-openflowplugin</artifactId>
 964                 <version>${openflowplugin.version}</version>
 965                 <classifier>features</classifier><type>xml</type>
 966         </dependency>
 967
 968 ``latest_snapshot`` in ``<openflowplugin.version>`` replace with latest snapshot, which can be found `here <https://nexus.opendaylight.org/content/repositories/opendaylight.snapshot/org/opendaylight/openflowplugin/openflowplugin/>`__.
 969
 970 | And in ``<topoprocessing_directory>/features/src/main/resources/features.xml``
 971 | add ``<repository>mvn:org.opendaylight.openflowplugin/features-openflowplugin/${openflowplugin.version}/xml/features</repository>``
 972   to repositories section.
 973
 974 Now after you rebuild project and start Karaf, you can install necessary features.
 975
 976 | You can install all with one command:
 977 | ``feature:install odl-restconf-noauth odl-topoprocessing-inventory-rendering odl-openflowplugin-southbound odl-openflowplugin-nsf-model``
 978
 979 Now you can send messages to REST from any REST client (e.g. Postman in
 980 Chrome). Messages have to have following headers:
 981
 982 +--------------------------------------+--------------------------------------+
 983 | Header                               | Value                                |
 984 +======================================+======================================+
 985 | Content-Type:                        | application/xml                      |
 986 +--------------------------------------+--------------------------------------+
 987 | Accept:                              | application/xml                      |
 988 +--------------------------------------+--------------------------------------+
 989 | username:                            | admin                                |
 990 +--------------------------------------+--------------------------------------+
 991 | password:                            | admin                                |
 992 +--------------------------------------+--------------------------------------+
 993
 994 Firstly send topology request to
 995 http://localhost:8181/restconf/config/network-topology:network-topology/topology/render:1
 996 with method PUT. Example of simple rendering request:
 997
 998 .. code:: xml
 999
1000     <topology xmlns="urn:TBD:params:xml:ns:yang:network-topology">
1001       <topology-id>render:1</topology-id>
1002         <correlations xmlns="urn:opendaylight:topology:correlation" >
1003           <output-model>inventory-rendering-model</output-model>
1004           <correlation>
1005              <correlation-id>1</correlation-id>
1006               <type>rendering-only</type>
1007               <correlation-item>node</correlation-item>
1008               <rendering>
1009                 <underlay-topology>und-topo:1</underlay-topology>
1010             </rendering>
1011           </correlation>
1012         </correlations>
1013     </topology>
1014
1015 This request says that we want create topology with name render:1 and
1016 this topology should be stored in the inventory-rendering-model and it
1017 should be created from topology flow:1 by node rendering.
1018
1019 Next we send the network-topology part of topology flow:1. So to the URL
1020 http://localhost:8181/restconf/config/network-topology:network-topology/topology/und-topo:1
1021 we PUT:
1022
1023 .. code:: xml
1024
1025     <topology xmlns="urn:TBD:params:xml:ns:yang:network-topology"
1026               xmlns:it="urn:opendaylight:model:topology:inventory"
1027               xmlns:i="urn:opendaylight:inventory">
1028         <topology-id>und-topo:1</topology-id>
1029         <node>
1030             <node-id>openflow:1</node-id>
1031             <it:inventory-node-ref>
1032         /i:nodes/i:node[i:id="openflow:1"]
1033             </it:inventory-node-ref>
1034             <termination-point>
1035                 <tp-id>tp:1</tp-id>
1036                 <it:inventory-node-connector-ref>
1037                     /i:nodes/i:node[i:id="openflow:1"]/i:node-connector[i:id="openflow:1:1"]
1038                 </it:inventory-node-connector-ref>
1039             </termination-point>
1040         </node>
1041     </topology>
1042
1043 And the last input will be inventory part of topology. To the URL
1044 http://localhost:8181/restconf/config/opendaylight-inventory:nodes we
1045 PUT:
1046
1047 .. code:: xml
1048
1049     <nodes
1050         xmlns="urn:opendaylight:inventory">
1051         <node>
1052             <id>openflow:1</id>
1053             <node-connector>
1054                 <id>openflow:1:1</id>
1055                 <port-number
1056                     xmlns="urn:opendaylight:flow:inventory">1
1057                 </port-number>
1058                 <current-speed
1059                     xmlns="urn:opendaylight:flow:inventory">10000000
1060                 </current-speed>
1061                 <name
1062                     xmlns="urn:opendaylight:flow:inventory">s1-eth1
1063                 </name>
1064                 <supported
1065                     xmlns="urn:opendaylight:flow:inventory">
1066                 </supported>
1067                 <current-feature
1068                     xmlns="urn:opendaylight:flow:inventory">copper ten-gb-fd
1069                 </current-feature>
1070                 <configuration
1071                     xmlns="urn:opendaylight:flow:inventory">
1072                 </configuration>
1073                 <peer-features
1074                     xmlns="urn:opendaylight:flow:inventory">
1075                 </peer-features>
1076                 <maximum-speed
1077                     xmlns="urn:opendaylight:flow:inventory">0
1078                 </maximum-speed>
1079                 <advertised-features
1080                     xmlns="urn:opendaylight:flow:inventory">
1081                 </advertised-features>
1082                 <hardware-address
1083                     xmlns="urn:opendaylight:flow:inventory">0E:DC:8C:63:EC:D1
1084                 </hardware-address>
1085                 <state
1086                     xmlns="urn:opendaylight:flow:inventory">
1087                     <link-down>false</link-down>
1088                     <blocked>false</blocked>
1089                     <live>false</live>
1090                 </state>
1091                 <flow-capable-node-connector-statistics
1092                     xmlns="urn:opendaylight:port:statistics">
1093                     <receive-errors>0</receive-errors>
1094                     <receive-frame-error>0</receive-frame-error>
1095                     <receive-over-run-error>0</receive-over-run-error>
1096                     <receive-crc-error>0</receive-crc-error>
1097                     <bytes>
1098                         <transmitted>595</transmitted>
1099                         <received>378</received>
1100                     </bytes>
1101                     <receive-drops>0</receive-drops>
1102                     <duration>
1103                         <second>28</second>
1104                         <nanosecond>410000000</nanosecond>
1105                     </duration>
1106                     <transmit-errors>0</transmit-errors>
1107                     <collision-count>0</collision-count>
1108                     <packets>
1109                         <transmitted>7</transmitted>
1110                         <received>5</received>
1111                     </packets>
1112                     <transmit-drops>0</transmit-drops>
1113                 </flow-capable-node-connector-statistics>
1114             </node-connector>
1115             <node-connector>
1116                 <id>openflow:1:LOCAL</id>
1117                 <port-number
1118                     xmlns="urn:opendaylight:flow:inventory">4294967294
1119                 </port-number>
1120                 <current-speed
1121                     xmlns="urn:opendaylight:flow:inventory">0
1122                 </current-speed>
1123                 <name
1124                     xmlns="urn:opendaylight:flow:inventory">s1
1125                 </name>
1126                 <supported
1127                     xmlns="urn:opendaylight:flow:inventory">
1128                 </supported>
1129                 <current-feature
1130                     xmlns="urn:opendaylight:flow:inventory">
1131                 </current-feature>
1132                 <configuration
1133                     xmlns="urn:opendaylight:flow:inventory">
1134                 </configuration>
1135                 <peer-features
1136                     xmlns="urn:opendaylight:flow:inventory">
1137                 </peer-features>
1138                 <maximum-speed
1139                     xmlns="urn:opendaylight:flow:inventory">0
1140                 </maximum-speed>
1141                 <advertised-features
1142                     xmlns="urn:opendaylight:flow:inventory">
1143                 </advertised-features>
1144                 <hardware-address
1145                     xmlns="urn:opendaylight:flow:inventory">BA:63:87:0C:76:41
1146                 </hardware-address>
1147                 <state
1148                     xmlns="urn:opendaylight:flow:inventory">
1149                     <link-down>false</link-down>
1150                     <blocked>false</blocked>
1151                     <live>false</live>
1152                 </state>
1153                 <flow-capable-node-connector-statistics
1154                     xmlns="urn:opendaylight:port:statistics">
1155                     <receive-errors>0</receive-errors>
1156                     <receive-frame-error>0</receive-frame-error>
1157                     <receive-over-run-error>0</receive-over-run-error>
1158                     <receive-crc-error>0</receive-crc-error>
1159                     <bytes>
1160                         <transmitted>576</transmitted>
1161                         <received>468</received>
1162                     </bytes>
1163                     <receive-drops>0</receive-drops>
1164                     <duration>
1165                         <second>28</second>
1166                         <nanosecond>426000000</nanosecond>
1167                     </duration>
1168                     <transmit-errors>0</transmit-errors>
1169                     <collision-count>0</collision-count>
1170                     <packets>
1171                         <transmitted>6</transmitted>
1172                         <received>6</received>
1173                     </packets>
1174                     <transmit-drops>0</transmit-drops>
1175                 </flow-capable-node-connector-statistics>
1176             </node-connector>
1177             <serial-number
1178                 xmlns="urn:opendaylight:flow:inventory">None
1179             </serial-number>
1180             <manufacturer
1181                 xmlns="urn:opendaylight:flow:inventory">Nicira, Inc.
1182             </manufacturer>
1183             <hardware
1184                 xmlns="urn:opendaylight:flow:inventory">Open vSwitch
1185             </hardware>
1186             <software
1187                 xmlns="urn:opendaylight:flow:inventory">2.1.3
1188             </software>
1189             <description
1190                 xmlns="urn:opendaylight:flow:inventory">None
1191             </description>
1192             <ip-address
1193                 xmlns="urn:opendaylight:flow:inventory">10.20.30.40
1194           </ip-address>
1195             <meter-features
1196                 xmlns="urn:opendaylight:meter:statistics">
1197                 <max_bands>0</max_bands>
1198                 <max_color>0</max_color>
1199                 <max_meter>0</max_meter>
1200             </meter-features>
1201             <group-features
1202                 xmlns="urn:opendaylight:group:statistics">
1203                 <group-capabilities-supported
1204                     xmlns:x="urn:opendaylight:group:types">x:chaining
1205                 </group-capabilities-supported>
1206                 <group-capabilities-supported
1207                     xmlns:x="urn:opendaylight:group:types">x:select-weight
1208                 </group-capabilities-supported>
1209                 <group-capabilities-supported
1210                     xmlns:x="urn:opendaylight:group:types">x:select-liveness
1211                 </group-capabilities-supported>
1212                 <max-groups>4294967040</max-groups>
1213                 <actions>67082241</actions>
1214                 <actions>0</actions>
1215             </group-features>
1216         </node>
1217     </nodes>
1218
1219 After this, the expected result from a GET request to
1220 http://127.0.0.1:8181/restconf/operational/network-topology:network-topology
1221 is:
1222
1223 .. code:: xml
1224
1225     <network-topology
1226         xmlns="urn:TBD:params:xml:ns:yang:network-topology">
1227         <topology>
1228             <topology-id>render:1</topology-id>
1229             <node>
1230                 <node-id>openflow:1</node-id>
1231                 <node-augmentation
1232                     xmlns="urn:opendaylight:topology:inventory:rendering">
1233                     <ip-address>10.20.30.40</ip-address>
1234                     <serial-number>None</serial-number>
1235                     <manufacturer>Nicira, Inc.</manufacturer>
1236                     <description>None</description>
1237                     <hardware>Open vSwitch</hardware>
1238                     <software>2.1.3</software>
1239                 </node-augmentation>
1240                 <termination-point>
1241                     <tp-id>openflow:1:1</tp-id>
1242                     <tp-augmentation
1243                         xmlns="urn:opendaylight:topology:inventory:rendering">
1244                         <hardware-address>0E:DC:8C:63:EC:D1</hardware-address>
1245                         <current-speed>10000000</current-speed>
1246                         <maximum-speed>0</maximum-speed>
1247                         <name>s1-eth1</name>
1248                     </tp-augmentation>
1249                 </termination-point>
1250                 <termination-point>
1251                     <tp-id>openflow:1:LOCAL</tp-id>
1252                     <tp-augmentation
1253                         xmlns="urn:opendaylight:topology:inventory:rendering">
1254                         <hardware-address>BA:63:87:0C:76:41</hardware-address>
1255                         <current-speed>0</current-speed>
1256                         <maximum-speed>0</maximum-speed>
1257                         <name>s1</name>
1258                     </tp-augmentation>
1259                 </termination-point>
1260             </node>
1261         </topology>
1262     </network-topology>
1263
1264 Use Cases
1265 ---------
1266
1267 You can find use case examples on `this wiki page
1268 <https://wiki.opendaylight.org/view/Topology_Processing_Framework:Developer_Guide:Use_Case_Tutorial>`__.
1269
1270 Key APIs and Interfaces
1271 -----------------------
1272
1273 The basic provider class is TopoProcessingProvider which provides
1274 startup and shutdown methods. Otherwise, the framework communicates via
1275 requests and outputs stored in the MD-SAL datastores.
1276
1277 API Reference Documentation
1278 ---------------------------
1279
1280 You can find API examples on `this wiki
1281 page <https://wiki.opendaylight.org/view/Topology_Processing_Framework:Developer_Guide:REST_API_Specification>`__.
1282