Include reports on previous releases as drafts.
Move ReST coala check to come before Python,
because the Python check takes longer to finish.
Change-Id: I84c34570c329f8793b76d07503db800999549da8
Signed-off-by: Vratko Polak <vrpolak@cisco.com>
Signed-off-by: Peter Gubka <pgubka@cisco.com>
http://,
https://
+[reStructuredText]
+bears = RSTcheckBear
+files = **.rst
+ignore = .**
+
[Python]
bears = PEP8Bear
files = **/*.py
ignore = .*/**
max_line_length = 120
-
-[reStructuredText]
-bears = RSTcheckBear
-files = **.rst
-ignore = .**
--- /dev/null
+=======
+Caveats
+=======
+
+This sub-page describes ways the test implementation (or results) differs
+from the `original specification <scenarios.html>`_ and which information motivates the difference.
+
+Jenkins job structure
+~~~~~~~~~~~~~~~~~~~~~
+
++ Information
+
+At the start of test implementation, all the Controller 3node test cases were added into an existing Jenkins job.
+
+During test development it was become clear, that adding all possible tests would make the job to run too long.
+
+Dividing the job into several smaller ones is possible, but most likely the history would be lost,
+unless Linux Foundation admins figure out a way to create multiple job clones with history copied.
+
++ Testing consequence
+
+Even with number of test cases reduced (see below), the job duration is around three and half hours.
+
++ How to fix
+
+After Carbon SR2 release, the jobs can be split, as there will be enough time
+to generate new history till Carbon SR3.
+
+Akka bugs
+~~~~~~~~~
+
+These are bugs which need either a fix in Akka codebase,
+or a workaround which would be too time-consuming to implement in ODL.
+
+Both bugs manifest as UnreachableMember event (without intentional isolation).
+
+Slow heartbeats
+---------------
+
++ Information
+
+Akka sends periodic heartbeats in order to detect when the other member is being unresponsive.
+
+The heartbeats are being serialized into the same TCP channel as ordinary data,
+which means if ODL is processing big amount of data, the heartbeats can spend a long time
+in TCP (or other) buffers before being processed. When this time exceeds a specific value
+(currently 6 seconds), the peer memeber is declared unreachable, generally leading to leader movement.
+
+This affects BGP test results on 3node setup, as ODL is processing BGP data as quickly as possible,
+but the current BGP implementation does not handle rib owner movement gracefully (and leader movement
+is explicitly checked by the test, as the scenario dictates it should not happen).
+This does not affect other data broker tests, 1000 transactions per second do not generate critical throughput.
+
++ Testing consequence
+
+Three test cases are failing due to `Bug 8318 <https://bugs.opendaylight.org/show_bug.cgi?id=8318>`__.
+
++ How to fix
+
+Possibly, a different akka configuration could be applied to separate akka cluster status messages
+into a different TCP stream than ordinary data stream.
+
+Otherwise, a contribution to Akka project would be needed.
+
+Reachability gossip
+-------------------
+
++ Information
+
+Akka uses a gossip protocol to advertize one member's reachability to other members.
+There is a logic which allows for faster detection of unreachable members,
+when a member can declare its peer unreachable if it got information from another peer
+which is considered more up-to-date.
+
+Ocassionally, this logic results in undesired behavior. This is when the supposedly up-to-date peer
+has been isolated and now it is rejoining. Depending on timing, this can introduce additional leader movement,
+or a very brief moment when a member "forgets" RPC registrations from other member.
+
+This is causing bugs `8420 <https://bugs.opendaylight.org/show_bug.cgi?id=8420>`__
+and `8430 <https://bugs.opendaylight.org/show_bug.cgi?id=8430>`__.
+
++ Testing consequence
+
+This affects "partition and heal" scenarios in singleton testing.
+In functional tests, the failures are infrequent enough to consider the test mostly stable overall,
+but the corresponding longevity jobs are failing consistently.
+
+The tests for "partition and heal" scenarios in RPC testing have been changed
+to tolerate wrong RPC results for 10 seconds to work around this Akka bug.
+
++ How to fix
+
+This does not seem fixable on ODL level, contribution to Akka project is needed.
+
+Missing features
+~~~~~~~~~~~~~~~~
+
+Cluster yang notifications
+--------------------------
+
++ Information
+
+Yang notifications are not delivered to peer members.
+`Bug 2139 <https://bugs.opendaylight.org/show_bug.cgi?id=2139>`__
+is only fixed for data change notifications, not Yang notifications.
+
+`Bug 2140 <https://bugs.opendaylight.org/show_bug.cgi?id=2140>`__ tracks adding this missing functionality.
+
++ Testing consequence
+
+Notification suites are running on 1-node setup only.
+
++ How to fix
+
+After the funtionality is added, it will be straightforward to add 3node tests.
+
+New features
+~~~~~~~~~~~~
+
+Tell-based protocol
+-------------------
+
++ Information
+
+Tell-based protocol is an alternative to ask-based protocol from Boron.
+Which protocol to use is decided by a line in a configuration file
+(org.opendaylight.controller.cluster.datastore.cfg).
+
+Some scenarios are expected to fail due to known limitations of ask-based protocol.
+More specifically, if a shard leader moves while a transaction is open in ask-based protocol,
+the transaction will fail (AskTimeoutException).
+
+This affects only data broker tests, not RPC calls.
+
++ Testing consequence
+
+In principle, this doubles the number of configurations to be tested, but see below.
+
++ How to fix
+
+It is planned for tell-based protocol to become the default setting after Carbon SR2.
+After that, tests for ask-based protocol can be converted or removed.
+
+Prefix-based shards
+-------------------
+
++ Information
+
+Tell-based shards are an alternative to module-based shards from Boron.
+Tell-based shards can be only created dynamically (as opposed to being read from a configuration file at startup).
+It is possible to use both types of shards, but data writes and reads use different API,
+so any Mdsal application needs to know which API to use.
+
+The implementation of prefix-based shards is hardwired to tell-based protocol
+(even if ask-based protocol is configured as the default).
+
++ Testing consequence
+
+This doubles the number of configurations to be tested, for tests related to data droker (RPCs are unaffected).
+
++ How to fix
+
+ODL contains great many applications which use APIs for module-based shards.
+It is expected that multiple releases would still need both types of tests cases.
+Module-based shards will be deprecated and removed eventually.
+
+Producer options
+----------------
+
++ Information
+
+Data producers for module-based shards can produce either chained transactions or standalone transactions.
+Data producers for prefix-based shards can produce either non-isolated transactions (change notifications
+can combine several transactions together) or isolated transactions.
+
++ Testing consequence
+
+In principle, this results in multiple Robot test cases for the same documented scenario case, but see below.
+
++ How to fix
+
+All test cases will be needed in forseeable future.
+Instead, more negative test cases may need be added to verify different options lead to different behavior.
+
+Initial leader placement
+~~~~~~~~~~~~~~~~~~~~~~~~
+
++ Information
+
+Some scenarios do not specify initial locations of relevant shard leaders.
+Test results can depend on it in presence of bugs.
+
+This is mostly relevant to BGP test, which has three relevant members:
+Rib owner, default operation shard leader and topology operational shard leader.
+
++ Testing consequence
+
+Two test cases are tested. The two shard leaders are always together, rib owner is either co-located or not.
+This is done by suite moving shard leaders after detecting rib owner location.
+
++ How to fix
+
+Even more placements can be tested when job duration stops being the limiting factor.
+
+Reduced BGP scaling
+~~~~~~~~~~~~~~~~~~~
+
++ Information
+
+Rib owner maintains de-duplicated data structures.
+Other members get serialized copies and they do not de-duplicate.
+
+Even single node strugless to fit into 6GB heap with tell-based protocol,
+see `Bug 8649 <https://bugs.opendaylight.org/show_bug.cgi?id=8649>`__.
+
++ Testing consequence
+
+Scale from reported tests reduced from 1 million prefixes to 300 thousand prefixes.
+
++ How to fix
+
+Other members should be able to perform de-duplication, but developing that takes effort.
+
+In the meantime, Linux Foundation could be convinced to allow for bigger VMs,
+currently limited by infrastructure available.
+
+Increased timeouts
+~~~~~~~~~~~~~~~~~~
+
+RequestTimeoutException
+-----------------------
+
++ Information
+
+With tell-based protocol, restconf requests might stay open up to 120 seconds before returning an error.
+Even shard state reads using Jolokia can take long time if the shard actor is busy processing other messages.
+
++ Testing consequence
+
+This increases duration for tests which need to verify transaction errors do happen
+after sufficiently long isolation. Also, duration is increased if a test fails on a read which is otherwise quick.
+
++ How to fix
+
+This involves a trade-off between stability and responsiveness.
+As MD-SAL applications rarely tolerate transaction failures, users would prefer stability.
+That means relatively longer timeouts are there to stay, which means test case duration
+will stay high in negative (or failing positive) tests.
+
+Client abort timeout
+--------------------
+
++ Information
+
+Client abort timeout is currently set to 15 minutes. The operational consequence is
+just an inability to start another data producer on a member isolated for that long.
+This test has too long duration compared to its usefulness.
+
++ Testing consequence
+
+This test case has never been implemented.
+
+Instead a test with isolation shorter than 120 seconds is implemented,
+the test verifies the data producer continues its operation without RequestTimeoutException.
+
++ How to fix
+
+It is straighforward to add the missing test cases when job duration stops being a limiting factor.
+
+No shard shutdown
+~~~~~~~~~~~~~~~~~
+
++ Common information.
+
+There are multiple RPCs offering different "severity" of shard shutdown.
+For technical details see comments on `change 58580 <https://git.opendaylight.org/gerrit/58580>`__.
+
+If tests perform rigorous teardown, the shard replica should be re-activated,
+which is an operation not every RPC supports.
+
+Listener stability suite
+------------------------
+
++ Information
+
+Current implementation of data listeners relies on a shard replica to be active on a member
+which is to receive the notification. Until that is imroved,
+`Bug 8629 <https://bugs.opendaylight.org/show_bug.cgi?id=8629>`__ prevents this scenario
+from being tested as described.
+
++ Testing consequence
+
+The suite uses become-leader RPC instead. This has an added benefit of test case being able to pick which member
+is to become the new leader (adding one more test case when the old leader was not co-located with the listener).
+
+Also, no teardown step is needed, the final cluster state is not missing any shard replica.
+
++ How to fix
+
+The original test can be implemented when listener implementation changes.
+But the test which uses become-leader might be better overall.
+
+Clean leader shutdown suite
+---------------------------
+
++ Information
+
+Some implementations of shutdown RPCs have a side effect of also shutting down shard state notifier.
+For details see `Bug 8794 <https://bugs.opendaylight.org/show_bug.cgi?id=8794>`__.
+
+The remove-shard-replica RPC does not have this downside, but it changes shard configuration,
+which was not intended by the original scenario definition.
+
++ Testing consequence
+
+Test cases for this scenario were switched to use remove-shard-replica.
+
++ How to fix
+
+There is an open debate on whether "shard shutdown" RPC with less operations (compared to remove-shard-replica)
+is something user wants and should be given access to.
+
+If yes, tests can be switched to such an RPC, assuming the shard notifier issue is also fixed.
+
+Hard reboots between test cases
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
++ Information
+
+Timing errors in Robot code lead to Robot being unable to restore original state without restarts.
+
+During development, we started without any hard reboots, and that was finding bugs in teardown steps of scenarios.
+But test independence was more important at that time, so current tests are less sensitive to teardown failures.
+
++ Testing consequence
+
+Around 115 second per ODL reboot, this time is added to every test case running time.
+Together with increased timeouts, this motivates leaving out some test cases to allow faster change verification.
+
++ How to fix
+
+Ideally, we would want both jobs with hard resets and jobs without them.
+The jobs without resets can be added gradually after splitting the current single job.
+
+Isolation mechanics
+~~~~~~~~~~~~~~~~~~~
+
++ Information
+
+During development, it was found that freeze and kill mechanics affect the co-located java test driver
+without exposing any new bugs.
+
+Turns out AAA functionality attempts to read from datastore, so isolated member returns http status code 401.
+
++ Testing consequence
+
+Only iptables filtering is used in order to reduce test job duration.
+
+Isolated members are never queried directly. A leader member is considered isolated
+when other members elect a lew leader. A member is considered rejoined
+when it responds reporting itself as a follower.
+
++ How to fix
+
+It is straightforward to add test cases for kill and freeze where appropriate,
+but once again this can be done gradually when job duration is not a limiting factor.
+
+Reduced number of combinations
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
++ Information
+
+Prefix-based shards always use tell-based protocol, so suites which test them
+with ask-based protocol configuration can be skipped.
+
+Ask-based protocol is known to fail on AskTimeoutException on leader movement,
+so suites which produce transactions constantly can be skipped.
+
+Most test cases are not sensitive to data producer options.
+
++ Testing consequence
+
+BGP tests and singleton tests use module-based shards only, both protocols.
+Other suites related to data broker are testing only tell-based protocol, both shard types.
+Netconf tests and RPC tests use module-based shards with ask-based protocol only.
+Only client isolaton suite tests different producer options.
+
++ How to fix
+
+More ests can be added gradually (see above).
+
+Possibly, not every combination is worth the duration it takes,
+but that could be alleviated if Linux Foundation infrastructure grows in size significantly.
+
+Reduced performance
+~~~~~~~~~~~~~~~~~~~
+
++ Information
+
+In order to reduce test job duration, suites wait for minimal functionality (jolokia reporting shards are in sync)
+after restarting ODL. That means unrelated karaf features might still being installed
+whet test is in progress. This should not affect functional tests, but it can reduce performance observed.
+
+The only suite observing strong enough performance inpact is `chasing the leader`_.
+
++ Testing consequence
+
+Functional tests for `chasing the leader`_ suite tolerate frequencies higher than 50 un-registrations per second.
+Longevity suite still requires full 100 unregistrations per second.
+
++ How to fix
+
+Suite can wait for better symptom of ODL being ready, for example by requiring CPU usage to become less
+that a chosen threshold.
+
+Missing logs
+~~~~~~~~~~~~
+
++ Information
+
+Robot VM has only 2GB of RAM and longevity jobs tend to produce large output.xml files.
+
+Ocasionally, a job can create karaf.log files so large they fail to download,
+in extreme cases filling ODL VM disk and causing failures.
+
+This affects mostly longevity jobs (and runs with verbose logging) if they pass.
+
++ Testing consequence
+
+Robot data stored is reduced to avoid this issue, sometimes leading to less details available.
+This issue is still not fully resolved, so ocassionally Robot log or karaf log is still missing
+if the job in question fails in an unexpected way.
+
++ How to fix
+
+It is possible for Robot test to put additional data into separate files.
+Unnecessarily verbose logs could be fixed where needed.
+
+As this limitation only hurts in newly occuring bugs, it is not really possible to entirely avoid this.
+
+Weekend outages
+~~~~~~~~~~~~~~~
+
++ Information
+
+Linux foundation ifrastructure teem occasionally needs to perform changes which affect running jobs.
+To reduce this impact, such changes are usually done over weekend.
+
+Cluster testing currently contains seve longevity jobs which block resources for 23 hours.
+As that is a significant portion of available resources, the longevity jobs are only run on weekend
+where the impact on frequency of other job is less critical.
+
++ Testing consequence
+
+Sometimes, the longevity jobs are affected by infrastructure team activities,
+leading to lost results or spurious failures.
+One such symptom is tracked as `Bug 8959 <https://bugs.opendaylight.org/show_bug.cgi?id=8959>`__.
+
++ How to fix
+
+It might be possible to spread longevity jobs over work days. As distributing jobs manually
+is not a scalable option, a considerable work would be needed to create an automatic way.
+
+Infrastructure changes are not very frequent, and having jobs run at the same predictable time
+is convenient from reporting point of view, so perhaps it is okay to keep the current setup.
+
+.. _`chasing the leader`: scenarios.html#chasing-the-leader
--- /dev/null
+Carbon cluster testing
+======================
+
+Contents:
+
+.. toctree::
+ :maxdepth: 1
+
+ scenarios
+ caveats
+ tests
+ sandbox
+ release
+ sr1
+ sr2
+
+Note that `release <release.html>`_, `sr1 <sr1.html>`_ and `sandbox <sandbox.html>`_ pages
+contain data from before test implementation and documentation structure were finalized,
+so there may be inconsistencies.
+
+TODO: Re-test Carbon release and SR1 images (with retrofitted tests where needed)
+so users can see authoritative test results.
--- /dev/null
+
+Draft, outdated: Carbon release test report
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Table
+-----
+
+.. table:: Test results (pre-release)
+ :widths: 40,30,20,10
+
+ ================== ========== ================================================================= ======
+ Scenario name Run date Bug numbers Result
+ ================== ========== ================================================================= ======
+ bgp-1n-1m-a 2017-05-23 `PASS <https://logs.opendaylight.org/releng/jenkins092/bgpcep-csit-1node-periodic-bgp-ingest-only-carbon/290/archives/log.html.gz#s1-s2>`__
+ bgp-1n-1m-t 2017-05-23 `PASS <https://logs.opendaylight.org/releng/jenkins092/bgpcep-csit-1node-periodic-bgp-ingest-only-carbon/290/archives/log.html.gz#s1-s9>`__
+ bgp-3n-300k-ll-t 2017-05-23 `8318 <https://bugs.opendaylight.org/show_bug.cgi?id=8318#c10>`__ `FAIL <https://logs.opendaylight.org/releng/jenkins092/bgpcep-csit-3node-periodic-bgpclustering-only-carbon/290/archives/log.html.gz#s1-s2-t8-k2-k3-k7-k4-k1-k6-k1-k1-k1-k1-k1-k2-k1-k3-k1>`__
+ bgp-3n-300k-lr-t 2017-05-23 `8318 <https://bugs.opendaylight.org/show_bug.cgi?id=8318>`__ `FAIL <https://logs.opendaylight.org/releng/jenkins092/bgpcep-csit-3node-periodic-bgpclustering-only-carbon/290/archives/log.html.gz#s1-s4-t8-k2-k3-k7-k8-k1-k6-k1-k1-k1-k1-k1-k2-k1-k4>`__
+ ddb-cls-ms-ll-t 2017-05-23 `8403 <https://bugs.opendaylight.org/show_bug.cgi?id=8403#c9>`__ `FAIL <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/720/archives/log.html.gz#s1-s20-t1-k2-k9>`__
+ ddb-cls-ms-lr-t 2017-05-23 `PASS <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/720/archives/log.html.gz#s1-s20-t3>`__
+ ddb-cls-ps-ll-t 2017-05-23 `8403 <https://bugs.opendaylight.org/show_bug.cgi?id=8403#c9>`__ `FAIL <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/720/archives/log.html.gz#s1-s22-t1-k2-k9>`__
+ ddb-cls-ps-lr-t 2017-05-23 `PASS <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/720/archives/log.html.gz#s1-s22-t3>`__
+ ddb-elm-ms-lr-t 2017-05-23 `8403 <https://bugs.opendaylight.org/show_bug.cgi?id=8403#c6>`__ `FAIL <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/720/archives/log.html.gz#s1-s24-t1-k2-k10>`__
+ ddb-elm-ms-rr-t 2017-05-23 `PASS <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/720/archives/log.html.gz#s1-s24-t3>`__
+ ddb-elm-ms-rl-t 2017-05-23 `8403 <https://bugs.opendaylight.org/show_bug.cgi?id=8403#c6>`__ `FAIL <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/720/archives/log.html.gz#s1-s24-t5-k2-k10>`__
+ ddb-elm-ps-lr-t 2017-05-23 `PASS <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/720/archives/log.html.gz#s1-s26-t1>`__
+ ddb-elm-ps-rr-t 2017-05-23 `PASS <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/720/archives/log.html.gz#s1-s26-t3>`__
+ ddb-elm-ps-rl-t 2017-05-23 `8403 <https://bugs.opendaylight.org/show_bug.cgi?id=8403#c6>`__ `FAIL <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/720/archives/log.html.gz#s1-s26-t5-k2-k9>`__
+ ddb-li-ms-st-t 2017-05-23 `8445 <https://bugs.opendaylight.org/show_bug.cgi?id=8445#c3>`__ `FAIL <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/720/archives/log.html.gz#s1-s28-t1-k2-k26-k1-k2-k3-k1-k1>`__
+ ddb-li-ms-dt-t 2017-05-23 `8494 <https://bugs.opendaylight.org/show_bug.cgi?id=8494#c2>`__ `FAIL <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/720/archives/log.html.gz#s1-s28-t3-k2-k26-k1-k3>`__
+ ddb-li-ps-st-t 2017-05-23 `8371 <https://bugs.opendaylight.org/show_bug.cgi?id=8371#c6>`__ `FAIL <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/720/archives/log.html.gz#s1-s30-t1-k2-k26-k1-k2-k1-k1-k1>`__
+ ddb-li-ps-dt-t 2017-05-23 `8371 <https://bugs.opendaylight.org/show_bug.cgi?id=8371#c6>`__ `FAIL <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/720/archives/log.html.gz#s1-s30-t3-k2-k26-k1-k1>`__
+ ddb-ci-ms-ll-ct-t 2017-05-23 `8494 <https://bugs.opendaylight.org/show_bug.cgi?id=8494#c3>`__ `FAIL <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/720/archives/log.html.gz#s1-s32-t1-k2-k19-k1-k1>`__
+ ddb-ci-ms-ll-st-t 2017-05-23 `8494 <https://bugs.opendaylight.org/show_bug.cgi?id=8494#c3>`__ `FAIL <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/720/archives/log.html.gz#s1-s32-t3-k2-k19-k1-k1>`__
+ ddb-ci-ms-lr-ct-t 2017-05-23 `PASS <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/720/archives/log.html.gz#s1-s32-t5>`__
+ ddb-ci-ms-lr-st-t 2017-05-23 `PASS <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/720/archives/log.html.gz#s1-s32-t7>`__
+ ddb-ci-ps-ll-ct-t 2017-05-23 `8494 <https://bugs.opendaylight.org/show_bug.cgi?id=8494#c4>`__ `FAIL <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/720/archives/log.html.gz#s1-s34-t1-k2-k19-k1-k1>`__
+ ddb-ci-ps-ll-st-t 2017-05-23 `8494 <https://bugs.opendaylight.org/show_bug.cgi?id=8494#c4>`__ `FAIL <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/720/archives/log.html.gz#s1-s34-t3-k2-k19-k1-k1>`__
+ ddb-ci-ps-lr-ct-t 2017-05-23 `PASS <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/720/archives/log.html.gz#s1-s34-t5>`__
+ ddb-ci-ps-lr-st-t 2017-05-23 `PASS <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/720/archives/log.html.gz#s1-s34-t7>`__
+ ddb-ls-ms-ll-t 2017-05-23 `8524 <https://bugs.opendaylight.org/show_bug.cgi?id=8524#c1>`__ `FAIL <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/720/archives/log.html.gz#s1-s36-t1-k2-k12-k1-k3-k1>`__
+ ddb-ls-ms-lr-t 2017-05-23 `8534 <https://bugs.opendaylight.org/show_bug.cgi?id=8534>`__ `FAIL <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/720/archives/log.html.gz#s1-s36-t3-k2-k13-k1>`__
+ ddb-ls-ps-ll-t 2017-05-23 `8524 <https://bugs.opendaylight.org/show_bug.cgi?id=8524#c1>`__ `FAIL <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/720/archives/log.html.gz#s1-s38-t1-k2-k13-k1-k3-k1>`__
+ ddb-ls-ps-lr-t 2017-05-23 `8524 <https://bugs.opendaylight.org/show_bug.cgi?id=8524#c2>`__ `FAIL <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/720/archives/log.html.gz#s1-s38-t3-k2-k12-k1-k3-k1>`__
+ drb-rpp-ms-a 2017-05-23 `PASS <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/720/archives/log.html.gz#s1-s2>`__
+ drb-rph-ms-a 2017-05-23 `PASS <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/720/archives/log.html.gz#s1-s4>`__
+ drb-app-ms-a 2017-05-23 `PASS <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/720/archives/log.html.gz#s1-s6>`__
+ drb-aph-ms-a 2017-05-23 `PASS <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/720/archives/log.html.gz#s1-s8>`__
+ dnb-1n-60k-a 2017-05-23 `PASS <https://logs.opendaylight.org/releng/jenkins092/controller-csit-1node-rest-cars-perf-only-carbon/605/archives/log.html.gz#s1-s2>`__
+ ss-ms-ms-a 2017-05-23 `PASS <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/720/archives/log.html.gz#s1-s10>`__
+ ss-ph-ms-a 2017-05-23 `PASS <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/720/archives/log.html.gz#s1-s12>`__
+ ss-cl-ms-a 2017-05-23 `PASS <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/720/archives/log.html.gz#s1-s14>`__
+ ss-ms-ms-t 2017-05-23 `PASS <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/720/archives/log.html.gz#s1-s40>`__
+ ss-ph-ms-t 2017-05-23 `PASS <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/720/archives/log.html.gz#s1-s42>`__
+ ss-cl-ms-t 2017-05-23 `PASS <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/720/archives/log.html.gz#s1-s44>`__
+ netconf-ba-ms-a 2017-05-23 `PASS <https://logs.opendaylight.org/releng/jenkins092/netconf-csit-3node-clustering-only-carbon/554/archives/log.html.gz#s1-s2>`__
+ netconf-ok-ms-a 2017-05-23 `PASS <https://logs.opendaylight.org/releng/jenkins092/netconf-csit-3node-clustering-only-carbon/554/archives/log.html.gz#s1-s5>`__
+ netconf-rr-ms-a 2017-05-23 `PASS <https://logs.opendaylight.org/releng/jenkins092/netconf-csit-3node-clustering-only-carbon/554/archives/log.html.gz#s1-s7>`__
+ bgp-3n-300k-t-long 2017-05-14 `8443 <https://bugs.opendaylight.org/show_bug.cgi?id=8443>`__ `FAIL <https://logs.opendaylight.org/releng/jenkins092/bgpcep-csit-3node-bgpclustering-longevity-only-carbon/3/archives/log.html.gz#s1-s2-t1-k3-k1-k3-k1-k1-k1-k1-k1-k2-k1>`__
+ ddb-elm-mc-a-long 2017-05-14 `8434 <https://bugs.opendaylight.org/show_bug.cgi?id=8434>`__ `FAIL <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-ddb-expl-lead-movement-longevity-only-carbon/4/archives/log.html.gz#s1-t1-k2-k1-k1-k1-k1-k1-k1-k2-k1-k1-k2-k6-k1-k1-k1-k6-k2-k1-k2-k1-k1-k3-k3-k1>`__
+ drb-rpp-ms-a-long 2017-05-14 `PASS <https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-3node-drb-precedence-longevity-only-carbon/6/console>`__
+ drb-rph-ms-a-long 2017-05-14 `PASS <https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-3node-drb-partnheal-longevity-only-carbon/9/console>`__
+ dnb-1n-60k-a-long 2017-05-14 `PASS <https://logs.opendaylight.org/releng/jenkins092/controller-csit-1node-notifications-longevity-only-carbon/11/console.log.gz>`__
+ ss-ph-ms-a-long 2017-05-14 `8420 <https://bugs.opendaylight.org/show_bug.cgi?id=8420#c5>`__ `FAIL <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-cs-partnheal-longevity-only-carbon/5/archives/log.html.gz#s1-t1-k3-k1-k1-k1-k1-k1-k1-k2-k1-k1-k5-k3-k1-k2>`__
+ ss-cl-ms-a-long 2017-05-14 `PASS <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-cs-chasing-leader-longevity-only-carbon/4/archives/log.html.gz#s1>`__
+ ================== ========== ================================================================= ======
+
+For descriptions of test cases, see `description page <tests.html>`_.
+Note that the link contains current description,
+the details might have been implemented differently at SR1 release.
--- /dev/null
+
+Permanent draft, inaccessible: Sandbox test report
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Test Case Summary
+-----------------
+
+RelEng stability summary.
+
++ tba: Recent failures to be analyzed yet: 0.
++ test: Recent failures caused by wrong assumptions in test: 0.
++ akka: Recent failures related to pure UnreachableMember: 4.
++ tell: Recent failures not clearly caused by UnreachableMember: 6.
++ few: Tests passing unless low frequency failure happens: 2 (1 without duplication).
+ (Low frequency means UnreachableMemeber or similar,
+ related to Akka where Controller code has not real control.)
++ pass: Tests passing consistently: 41 (39 without duplication).
++ Total: 53 (50 without duplication).
++ Total minus akka: 49 (46 without duplication).
++ Total minus akka passing always or mostly: 43 (40 without duplication).
++ Acceptance rate: 43/49=87.75% (40/46=86.95% without duplication).
+
+Table
+-----
+
+S017 instead of 2017 means Sandbox run (includes changes not merged to stable/carbon yet).
+
+Last fail is date of last failure not caused by infra
+(or by a typo in test or by netconf/bgp failing to initialize properly).
+
+"S 17" or "2 17" in Last run means the documented run was superseded by a newer one, but not analyzed yet.
+
+"no sr3" means this test was not run on Sandbox, SR2 result is reported instead.
+"few" status from SR2 is not inherited (such tests are marked as "pass").
+"long ago" means the last real test failue happened somewhere before SR2 release (or never).
+
+TODO: Copy formatting from sr2 page.
+
+.. table:: Releng stability results (pre-SR2)
+ :widths: 30,10,20,20,10,10
+
+ ================== ===== ========== ========== ============================================================= ==========
+ Scenario name Type Last fail Last run Bugs Robot link
+ ================== ===== ========== ========== ============================================================= ==========
+ bgp-1n-1m-a pass no sr3 no sr3 no sr3
+ bgp-1n-300k-t pass no sr3 no sr3 no sr3
+ bgp-3n-300k-ll-t akka no sr3 no sr3 `8318 <https://bugs.opendaylight.org/show_bug.cgi?id=8318>`__ no sr3
+ bgp-3n-300k-lr-t akka no sr3 no sr3 `8318 <https://bugs.opendaylight.org/show_bug.cgi?id=8318>`__ no sr3
+ ddb-cls-ms-ll-t pass long ago S017-08-24 no fail this week
+ ddb-cls-ms-lr-t pass long ago S017-08-24 no fail this week
+ ddb-cls-ps-ll-t pass long ago S017-08-24 no fail this week
+ ddb-cls-ps-lr-t pass long ago S017-08-24 no fail this week
+ ddb-elm-ms-lr-t pass long ago S017-08-24 no fail this week
+ ddb-elm-ms-rr-t pass long ago S017-08-24 no fail this week
+ ddb-elm-ms-rl-t pass long ago S017-08-24 no fail this week
+ ddb-elm-ps-lr-t pass long ago S017-08-24 no fail this week
+ ddb-elm-ps-rr-t pass long ago S017-08-24 no fail this week
+ ddb-elm-ps-rl-t pass long ago S017-08-24 no fail this week
+ ddb-li-ms-st-t pass long ago S017-08-24 no fail this week
+ ddb-li-ms-dt-t pass long ago S017-08-24 no fail this week
+ ddb-li-ps-st-t pass long ago S017-08-24 no fail this week
+ ddb-li-ps-dt-t tell S017-08-24 S017-08-24 `8845 <https://bugs.opendaylight.org/show_bug.cgi?id=8845>`__ `link <https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon/2/log.html.gz#s1-s30-t3-k2-k25-k1-k8>`__
+ ddb-ci-ms-ll-ct-t pass long ago S017-08-24 no fail this week
+ ddb-ci-ms-ll-st-t pass long ago S017-08-24 no fail this week
+ ddb-ci-ms-lr-ct-t pass long ago S017-08-24 no fail this week
+ ddb-ci-ms-lr-st-t pass long ago S017-08-24 no fail this week
+ ddb-ci-ps-ll-ct-t pass long ago S017-08-24 no fail this week
+ ddb-ci-ps-ll-st-t pass long ago S017-08-24 no fail this week
+ ddb-ci-ps-lr-ct-t pass long ago S017-08-24 no fail this week
+ ddb-ci-ps-lr-st-t pass long ago S017-08-24 no fail this week
+ ddb-ls-ms-lr-t pass long ago S017-08-24 no fail this week
+ ddb-ls-ms-rr-t pass long ago S017-08-24 no fail this week
+ ddb-ls-ms-rl-t pass long ago S017-08-24 no fail this week
+ ddb-ls-ps-lr-t tell S017-08-24 S017-08-24 `8733 <https://bugs.opendaylight.org/show_bug.cgi?id=8733>`__ `link <https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon/2/log.html.gz#s1-s38-t1-k2-k14-k2-k1-k4-k7-k1>`__
+ ddb-ls-ps-rr-t tell S017-08-24 S017-08-24 `8733 <https://bugs.opendaylight.org/show_bug.cgi?id=8733>`__ `link <https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon/2/log.html.gz#s1-s38-t3-k2-k14-k2-k1-k4-k7-k1>`__
+ ddb-ls-ps-rl-t pass long ago S017-08-24 no fail this week
+ drb-rpp-ms-a pass long ago S017-08-24 no fail this week
+ drb-rph-ms-a pass long ago S017-08-24 no fail this week
+ drb-app-ms-a pass long ago S017-08-24 no fail this week
+ drb-aph-ms-a pass long ago S017-08-24 no fail this week
+ dnb-1n-60k-a pass no sr3 no sr3 no sr3
+ ss-ms-ms-a pass long ago S017-08-24 no fail this week
+ ss-ph-ms-a few S017-08-24 S017-08-24 `8420 <https://bugs.opendaylight.org/show_bug.cgi?id=8420>`__ `link <https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon/2/log.html.gz#s1-s10-t5-k2-k1-k1-k4>`__
+ ss-cl-ms-a pass long ago S017-08-24 no fail this week
+ ss-ms-ms-t pass long ago S017-08-24 no fail this week
+ ss-ph-ms-t few S017-08-24 S017-08-24 `8420 <https://bugs.opendaylight.org/show_bug.cgi?id=8420>`__ `link <https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon/2/log.html.gz#s1-s40-t5-k2-k1-k1-k4>`__
+ ss-cl-ms-t pass long ago S017-08-24 no fail this week
+ netconf-ba-ms-a pass no sr3 no sr3 no fail this week
+ netconf-ok-ms-a tell no sr3 no sr3 `9027 <https://bugs.opendaylight.org/show_bug.cgi?id=9027>`__ no fail this week
+ netconf-rr-ms-a tell no sr3 no sr3 `9027 <https://bugs.opendaylight.org/show_bug.cgi?id=9027>`__ no fail this week
+ bgp-3n-300k-t-long akka no sr3 no sr3 `8318 <https://bugs.opendaylight.org/show_bug.cgi?id=8318>`__ no sr3
+ ddb-elm-mc-t-long pass no sr3 no sr3 no sr3
+ drb-rpp-ms-a-long pass no sr3 no sr3 no sr3
+ drb-rph-ms-a-long pass no sr3 no sr3 `8430 <https://bugs.opendaylight.org/show_bug.cgi?id=8430>`__ no sr3
+ dnb-1n-60k-a-long pass no sr3 no sr3 no sr3
+ ss-ph-ms-a-long akka no sr3 no sr3 `8420 <https://bugs.opendaylight.org/show_bug.cgi?id=8420>`__ no sr3
+ ss-cl-ms-a-long tell S017-08-23 S017-08-23 `9054 <https://bugs.opendaylight.org/show_bug.cgi?id=9054>`__ `link <https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-cs-chasing-leader-longevity-only-carbon/14/log.html.gz#s1-s2-t3-k3-k2-k1-k1-k2-k1-k4-k6-k1>`__
+ ================== ===== ========== ========== ============================================================= ==========
+
+For descriptions of test cases, see `description page <tests.html>`_.
+Note that the link contains current description,
+the details might have been implemented differently at SR1 release.
--- /dev/null
+
+Description of test scenarios
+:::::::::::::::::::::::::::::
+
+This is a test plan written around M1 of Carbon cycle.
+
+During the cycle several limitations were found,
+which resulted in tests which implement the scenarios
+is ways different from what is described here.
+
+For list of limitations and differences, see `caveats page <caveats.html>`_.
+For more detailed descriptions of test cases as implemented, see `test description page <tests.html>`_.
+
+Controller Cluster Service Functional Tests
+===========================================
+The purpose of functional tests is to establish a known baseline behavior
+for basic services exposed to application plugins when the cluster member nodes encounter problems.
+
+Isolation Mechanics
+ Three-node scenarios executed in tests below need to be repeated for three distinct modes of isolation:
+
+ 1) JVM freeze, initiated by 'kill -STOP <pid>' on the JVM process,
+ followed by a 'kill -CONT <pid>' after three minutes. This simulates
+ a long-running garbage collection cycle, VM suspension or similar,
+ after which the JVM recovers without losing state and scheduled timers going off simultaneously.
+ 2) Network-level isolation via firewalling. Simulates a connectivity issue between member nodes,
+ while all nodes continue to work as usual. This should be done
+ by firewalling all traffic to and from the target node.
+ 3) JVM restart. This simulates a hard error, such as JVM error, VM reboot, and similar.
+ The JVM loses its state and the scenario tests whether the failed node
+ is able to result its operations as a member of the cluster.
+
+Leader Shutdown
+ The Shard implementation allows a leader to be shut down at run time,
+ which is expected to perform a clean hand over to a new leader, elected from the remaining shard members.
+
+DOMDataBroker
+^^^^^^^^^^^^^
+Also known as 'the datastore', provides MVCC transaction and data change notifications.
+
+Leader Stability
+----------------
+The goal is to ensure that a single-established shard does not flap,
+i.e. does not trigger leader movement by causing crashes or timeouts.
+This is performed by having the BGP load generator
+run injection of 1 million prefixes, followed by their removal.
+
+This test is executed in three scenarios:
+
++ Single node
++ Three-node, with shard leader being local
++ Three-node, with shard leader being remote
+
+Success criteria are:
+
++ Both injection and removal succeed
++ No transaction errors reported to the generator
++ No leader movement on the backend
+
+Clean Leader Shutdown
+---------------------
+The goal is to ensure that applications do not observe disruption
+when a shard leader is shut down cleanly. This is performed by having
+a steady-stream producer execute operations against the shard
+and then initiate leader shard shutdown, then the producer is shut down cleanly.
+
+This test is executed in two scenarios:
+
++ Three-node, with shard leader being local
++ Three-node, with shard leader being remote
+
+Success criteria are:
+
++ No transaction errors occur
++ Producer shuts down cleanly (i.e. all transactions complete successfully)
+
+Test tool: *test-transaction-producer*, running at 1K tps
+
++ Steady, configurable producer started with:
+
+ + A transaction chain
+ + Single transactions (note: these cannot overlap)
+
++ Configurable transaction rate (i.e. transactions-per-second)
++ Single-operation transactions
++ Random mix across 1M entries
+
+Explicit Leader Movement
+------------------------
+The goal is to ensure that applications do not observe disruption
+when a shard leader is moved as the result of explicit application request.
+This is performed by having a steady-stream producer execute operations
+against the shard and then initiate shard leader shutdown,
+then the producer is shut down cleanly.
+
+This test is executed in three scenarios:
+
++ Three-node, with shard leader being local and becoming remote
++ Three-node, with shard leader being remote and remaining remote
++ Three-node, with shard leader being remote and becoming local
+
+Success criteria are:
+
++ No transaction errors occur
++ Producer shuts down cleanly (i.e. all transactions complete successfully)
+
+Test tool: test-transaction-producer, running at 1K tps
+Test tool: *test-leader-mover*
+
++ Uses cds-dom-api to request shard movement
+
+Leader Isolation
+----------------
+The goal is to ensure the datastore succeeds in basic isolation/rejoin scenario,
+simulating either a network partition, or a prolonged GC pause.
+
+This test is executed in the following two scenarios:
+
++ Three-node, partition heals within TRANSACTION_TIMEOUT
++ Three-node, partition heals after 2*TRANSACTION_TIMEOUT
+
+Using following steps:
+
+1) Start test-transaction producer, running at 1K tps, non-overlapping, from all nodes to a single shard
+2) Isolate leader
+3) Wait for followers to initiate election
+4) Un-isolate leader
+5) Wait for partition to heal
+6) Restart failed producer
+
+Success criteria:
+
++ Followers win election in 3
++ No transaction failures occur if the partition is healed within TRANSACTION_TIMEOUT
++ Producer on old leader works normally after step 6)
+
+Test tool: test-transaction-producer
+
+Client Isolation
+----------------
+The purpose of this test is to ascertain that the failure modes of cds-access-client work as expected.
+This is performed by having a steady stream of transactions flowing from the frontend
+and isolating the node hosting the frontend from the rest of the cluster.
+
+This test is executed in one scenario:
+
++ Three node, test-transaction-producer running on a non-leader
++ Three node, test-transaction-producer running on the leader
+
+Success criteria:
+
++ After TRANSACTION_TIMEOUT failures occur
++ After HARD_TIMEOUT client aborts
+
+Test tool: test-transaction-producer
+
+Listener Isolation
+------------------
+The goal is to ensure listeners do no observe disruption when the leader moves.
+This is performed by having a steady stream of transactions
+being observed by the listeners and having the leader move.
+
+This test is executed in two scenarios:
+
++ Three node, test-transaction-listener running on the leader
++ Three node, test-transaction-listener running on a non-leader
+
+Using these steps:
+
++ Start the listener on target node
++ Start test-transaction-producer on each node, with 1K tps, non-overlapping data
++ Trigger shard movement by shutting down shard leader
++ Stop producers without erasing data
++ Stop listener
+
+Success criteria:
+
++ Listener-internal data tree has to match data stored in the data tree
+
+Test tool: *test-transaction-listener*
+
++ Subscribes a DTCL to multiple subtrees (as specified)
++ DTCL applies reported changes to an internal DataTree
+
+DOMRpcBroker
+^^^^^^^^^^^^
+Responsible for routing RPC requests to their implementations and routing responses back to the caller.
+
+RPC Provider Precedence
+-----------------------
+The aim is to establish that remote RPC implementations have lower priority
+than local ones, which is to say that any movement of RPCs on remote nodes
+does not affect routing as long as a local implementation is available.
+
+Test is executed only in a three-node scenario, using the following steps:
+
+1) Register an RPC implementation on each node
+2) Invoke RPC on each node
+3) Unregister implementation on one node
+4) Invoke RPC on that node
+5) Re-register implementation on than node
+6) Invoke RPC on that node
+
+Success criteria:
+
++ Invocation in steps 2) and 6) results in a response from local node
++ Invocation in step 4) results in a response from one of the other two nodes
+
+RPC Provider Partition and Heal
+-------------------------------
+This tests establishes that the RPC service operates correctly when faced with node failures.
+
+Test is executed only in a three-node scenario, using the following steps:
+
+1) Register an RPC implementation on two nodes
+2) Invoke RPC on each node
+3) Isolate one of the nodes where RPC is registered
+4) Invoke RPC on each node
+5) Un-isolate the node
+6) Invoke RPC on all nodes
+
+Success criteria:
+
++ Step 2) routes the RPC the node nearest node (local or remote)
++ Step 4) works, routing the RPC request to the implementation in the same partition
++ Step 6) routes the RPC the node nearest node (local or remote)
+
+Action Provider Precedence
+--------------------------
+The aim is to establish that remote action implementations have lower priority than local ones,
+which is to say that any movement of actions on remote nodes does not affect routing
+as long as a local implementation is available.
+
+Test is executed only in a three-node scenario, using the following steps:
+
+1) Register an action implementation on each node
+2) Invoke action on each node
+3) Unregister implementation on one node
+4) Invoke action on that node
+5) Re-register implementation on than node
+6) Invoke action on that node
+
+Success criteria:
+
++ Invocation in steps 2) and 6) results in a response from local node
++ Invocation in step 4) results in a response from one of the other two nodes
+
+Action Provider Partition and Heal
+----------------------------------
+This tests establishes that the RPC service for actions operates correctly when faced with node failures.
+
+Test is executed only in a three-node scenario, using the following steps:
+
+1) Register an action implementation on two nodes
+2) Invoke action on each node
+3) Isolate one of the nodes where RPC is registered
+4) Invoke action on each node
+5) Un-isolate the node
+6) Invoke action on all nodes
+
+Success criteria:
+
++ Step 2) routes the action request the node nearest node (local or remote)
++ Step 4) works, routing the action request to the implementation in the same partition
++ Step 6) routes the RPC the node nearest node (local or remote)
+
+DOMNotificationBroker
+^^^^^^^^^^^^^^^^^^^^^
+Provides routing of YANG notifications from publishers to subscribers.
+
+No-loss rate
+------------
+The purpose of this test is to determine the broker can forward messages without loss.
+We do this on a single-node setup by incrementally adding publishers and subscribers.
+
+This test is executed in one scenario:
+
++ Single-node
+
+Steps:
+
++ Start test-notification-subscriber
++ Start test-notification-publisher at 5K notifications/sec
++ Run for 5 minutes, verify no notifications lost
++ Add another pair of publisher/subscriber, repeat for rate of 60K notifications/sec
+
+Success criteria:
+
++ No notifications lost at rate of 60K notifications/sec
+
+Test tool: *test-notification-publisher*
+
++ Publishes notifications containing instance id and sequence number
++ Configurable rate (i.e. notifications-per-second)
+
+Test tool: *test-notification-subscriber*
+
++ Subscribes to specified notifications from publisher
++ Verifies notification sequence numbers
++ Records total number of notifications received and number of sequence errors
+
+Cluster Singleton
+^^^^^^^^^^^^^^^^^
+Cluster Singleton service is designed to ensure that
+only one instance of an application is registered globally in the cluster.
+
+Master Stability
+----------------
+The goal is to establish the service operates correctly in face of application registration changing
+without moving the active instance.
+
+The test is performed in a three-node cluster using following steps:
+
+1) Register candidate on each node
+2) Wait for master activation
+3) Remove non-master candidate,
+4) Wait one minute
+5) Restore the removed candidate
+
+Success criteria:
+
++ After step 2) there is exactly one master in the cluster
++ The master does not move to a different node for the duration of the test
+
+Partition and Heal
+------------------
+The goal is to establish the service operates correctly in face of node failures.
+
+The test is performed in a three-node cluster using following steps:
+
+1) Register candidate on each node
+2) Wait for master activation
+3) Isolate master node
+4) Wait two minutes
+5) Un-isolate (former) master node
+6) Wait one minute
+
+Success criteria:
+
++ After step 3), master instance is brought down on isolated node
++ During step 4) majority partition elects a new master
++ Until 5) occurs, old master remains deactivated
++ After 6) old master remains deactivated
+
+Chasing the Leader
+------------------
+This test aims to establish the service operates correctly
+when faced with rapid application transitions without having a stabilized application.
+
+This test is performed in a three-node setup using the following steps:
+
+1) Register a candidate on each node
+2) Wait for master activation
+3) Newly activated master unregisters itself
+4) Repeat 2
+
+Success criteria:
+
++ No failures occur for 5 minutes
++ Transition speed is at least 100 movements per second
+
+Controller Cluster Services Longevity Tests
+===========================================
+
+1) Run No-Loss Rate test for 24 hours. No message loss, instability or memory leaks may occur.
+2) Repeat Leader Stability test for 24 hours. No transaction failures, instability, leader movement or memory leaks may occur.
+3) Repeat Explicit Leader Movement test for 24 hours. No transaction failures, instability, leader movement or memory leaks may occur.
+4) Repeat RPC Provider Precedence test for 24 hours. No failures or memory leaks may occur.
+5) Repeat RPC partition and Heal test for 24 hours. No failures or memory leaks may occur.
+6) Repeat Chasing the Leader test for 24 hours. No memory leaks or failures may occur.
+7) Repeat Partition and Heal test for 24 hours. No memory leaks or failures may occur.
+
+NETCONF System Tests
+====================
+Netconf is an MD-SAL application, which listens to config datastore changes,
+registers a singleton for every configured device, instantiated singleton is updating device connection data
+in operational datastore, maintaining a mount point and handling access to the mounted device.
+
+Basic configuration and mount point access
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+No disruptions, ordinary netconf operation with restconf calls to different cluster members.
+
+Test is executed in a three-node scenario, using the following steps:
+
+1) Configure connection to test device on member-1.
+2) Create, update and delete data on the device using calls to member-2.
+3) Each state change confirmed by reading device data on member-3.
+4) De-configure the device connection.
+
+Success criteria:
+
++ All reads confirm data operations are applied correctly.
+
+Device owner killed
+^^^^^^^^^^^^^^^^^^^
+Killing current device owner leads to electing new owner. Operations are still applied.
+
+The test is performed in a three-node cluster using following steps:
+
+1) Configure connection to test device on member-1.
+2) Create data on the device using a call to member-2.
+3) Locate and kill the device owner member.
+4) Wait for a new owner to get elected.
+5) Update data on the device using a call to one of the surviving members.
+6) Restart the killed member.
+7) Update the data again using a call to the restarted member.
+
+Success criteria:
+
++ Each operation (including restart) is confirmed by reads on all members currently up.
+
+Rolling restarts
+^^^^^^^^^^^^^^^^
+Each member is restarted (start is waiting for cluster sync) in succession,
+this is to guarantee each Leader is affected.
+
+The test is performed in a three-node cluster using following steps:
+
+1) Configure connection to test device on member-1.
+2) Kill member-1.
+3) Create data on the device using a call to member-2.
+4) Start member-1.
+5) Kill member-2.
+6) Update data on the device using a call to member-3.
+7) Start member-2.
+8) Kill member-3.
+9) Delete data on the device using a call to member-1.
+10) Start member-3.
+
+Success criteria:
+
++ After every operation, reads on both living members confirm it was applied.
++ After every start, a read on the started node confirms it sees the device data from the previous operation.
--- /dev/null
+
+Draft, outdated: Carbon SR1 test report
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Test Case Summary
+-----------------
+
+RelEng stability summary.
+
++ tba: Recent failures to be analyzed yet: 0.
++ test: Recent failures caused by wrong assumptions in test: 0.
++ akka: Recent failures related to pure UnreachableMember: 5.
++ tell: Recent failures not clearly caused by UnreachableMember: 9.
++ few: Tests passing unless low frequency failure happens: 22 (21 without duplication).
+ (Low frequency means UnreachableMemeber or "Message was not delivered, dead letters encountered",
+ both are related to Akka where Controller code has not real control.)
++ pass: Tests passing consistently: 17 (15 without duplication).
++ Total: 53 (50 without duplication).
++ Total minus akka: 48 (45 without duplication).
++ Total minus akka passing always or mostly: 39 (36 without duplication).
++ Acceptance rate: 39/48=81.25% (36/45=80.00% without duplication).
+
+Table
+-----
+
+S017 instead of 2017 means Sandbox run (includes changes not merged to stable/carbon yet).
+
+Last fail is date of last failure not caused by infra
+(or by a typo in test or by netconf/bgp failing to initialize properly).
+
+"S 17" or "2 17" in Last run means the documented run was superseded by a newer one, but not analyzed yet.
+
+"long ago" means the last real test failue happened before around 2017-05-19, or never.
+
+.. table:: Releng stability results (pre-SR1)
+ :widths: 30,10,20,20,10,10
+
+ ================== ===== ========== ========== ============================================================= ==========
+ Scenario name Type Last fail Last run Bugs Robot link
+ ================== ===== ========== ========== ============================================================= ==========
+ bgp-1n-1m-a pass long ago 2017-07-14 `link <https://jenkins.opendaylight.org/releng/view/bgpcep/job/bgpcep-csit-1node-periodic-bgp-ingest-only-carbon/lastSuccessfulBuild/robot/bgpcep-bgp-ingest.txt/Singlepeer%20Prefixcount/>`__
+ bgp-1n-300k-t pass long ago 2017-07-14 `link <https://jenkins.opendaylight.org/releng/view/bgpcep/job/bgpcep-csit-1node-periodic-bgp-ingest-only-carbon/lastSuccessfulBuild/robot/bgpcep-bgp-ingest.txt/Singlepeer%20Pc%20Shm%20300Kroutes_1/>`__
+ bgp-3n-300k-ll-t akka 2017-07-14 2017-07-14 `8318 <https://bugs.opendaylight.org/show_bug.cgi?id=8318>`__ `link <https://logs.opendaylight.org/releng/jenkins092/bgpcep-csit-3node-periodic-bgpclustering-only-carbon/343/log.html.gz#s1-s2-t8-k2-k3-k7-k5-k1-k6-k1-k1-k1-k1-k1-k2-k1-k4>`__
+ bgp-3n-300k-lr-t akka 2017-07-13 2017-07-14 `8318 <https://bugs.opendaylight.org/show_bug.cgi?id=8318>`__ `link <https://logs.opendaylight.org/releng/jenkins092/bgpcep-csit-3node-periodic-bgpclustering-only-carbon/343/log.html.gz#s1-s4-t8-k2-k3-k7-k7-k1-k6-k1-k1-k1-k1-k1-k2-k1-k2-k4>`__
+ ddb-cls-ms-ll-t few 2017-07-04 2017-07-15 `8794 <https://bugs.opendaylight.org/show_bug.cgi?id=8794>`__ `link <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/771/log.html.gz#s1-s20-t1-k2-k8>`__
+ ddb-cls-ms-lr-t few 2017-07-08 2017-07-15 `8618 <https://bugs.opendaylight.org/show_bug.cgi?id=8618>`__ `link <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/776/log.html.gz#s1-s20-t3-k2-k8>`__
+ ddb-cls-ps-ll-t few 2017-07-09 2017-07-15 `8794 <https://bugs.opendaylight.org/show_bug.cgi?id=8794>`__ `link <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/777/log.html.gz#s1-s22-t1-k2-k8>`__
+ ddb-cls-ps-lr-t pass long ago 2017-07-15 `link <https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-3node-clustering-only-carbon/lastSuccessfulBuild/robot/controller-clustering.txt/Clean%20Leader%20Shutdown%20Prefbasedshard/Remote_Leader_Shutdown/>`__
+ ddb-elm-ms-lr-t few 2017-06-13 2017-07-15 `8618 <https://bugs.opendaylight.org/show_bug.cgi?id=8618>`__ `link <https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon/31/log.html.gz#s1-s24-t1-k2-k10>`__
+ ddb-elm-ms-rr-t few 2017-06-10 2017-07-15 `8618 <https://bugs.opendaylight.org/show_bug.cgi?id=8618>`__ `link <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/738/log.html.gz#s1-s24-t3-k2-k10>`__
+ ddb-elm-ms-rl-t few 2017-06-27 2017-07-15 `8749 <https://bugs.opendaylight.org/show_bug.cgi?id=8749>`__ `link <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/763/log.html.gz#s1-s24-t5-k2-k10>`__
+ ddb-elm-ps-lr-t few 2017-06-11 2017-07-15 `8664 <https://bugs.opendaylight.org/show_bug.cgi?id=8664>`__ `link <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/736/log.html.gz#s1-s26-t1-k2-k6-k3-k1-k4-k7-k1>`__
+ ddb-elm-ps-rr-t pass long ago 2017-07-15 `link <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/lastSuccessfulBuild/log.html.gz#s1-s26-t3>`__
+ ddb-elm-ps-rl-t few 2017-06-07 2017-07-15 `8403 <https://bugs.opendaylight.org/show_bug.cgi?id=8403>`__ `link <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/735/log.html.gz#s1-s26-t5-k2-k9>`__
+ ddb-li-ms-st-t tell 2017-07-15 2017-07-15 `8792 <https://bugs.opendaylight.org/show_bug.cgi?id=8792>`__ `link <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/783/log.html.gz#s1-s28-t1-k2-k25-k1-k1>`__
+ ddb-li-ms-dt-t tell 2017-07-15 2017-07-15 `8619 <https://bugs.opendaylight.org/show_bug.cgi?id=8619>`__ `link <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/783/log.html.gz#s1-s28-t3-k2-k25-k1-k8>`__
+ ddb-li-ps-st-t few 2017-06-08 2017-07-15 `8371 <https://bugs.opendaylight.org/show_bug.cgi?id=8371>`__ `link <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/736/log.html.gz#s1-s30-t1-k2-k25-k1-k1>`__
+ ddb-li-ps-dt-t tell 2017-07-15 2017-07-15 `8845 <https://bugs.opendaylight.org/show_bug.cgi?id=8845>`__ `link <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/783/log.html.gz#s1-s30-t3-k2-k25-k1-k8>`__
+ ddb-ci-ms-ll-ct-t few 2017-06-07 2017-07-15 `8494 <https://bugs.opendaylight.org/show_bug.cgi?id=8494>`__ `link <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/735/log.html.gz#s1-s32-t1-k2-k16-k1-k1>`__
+ ddb-ci-ms-ll-st-t tell 2017-07-15 2017-07-15 `8494 <https://bugs.opendaylight.org/show_bug.cgi?id=8494>`__ `link <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/783/log.html.gz#s1-s32-t3-k2-k16-k1-k1>`__
+ ddb-ci-ms-lr-ct-t few 2017-06-08 2017-07-15 `8636 <https://bugs.opendaylight.org/show_bug.cgi?id=8636>`__ `link <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/736/log.html.gz#s1-s32-t5-k2-k15-k1-k1-k1-k1-k1-k1-k2-k1-k1-k1>`__
+ ddb-ci-ms-lr-st-t tell 2017-07-15 2017-07-15 `8494 <https://bugs.opendaylight.org/show_bug.cgi?id=8494>`__ `link <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/783/log.html.gz#s1-s32-t7-k2-k16-k1-k1>`__
+ ddb-ci-ps-ll-ct-t few 2017-06-28 2017-07-15 `8494 <https://bugs.opendaylight.org/show_bug.cgi?id=8494>`__ `link <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/764/log.html.gz#s1-s34-t1-k2-k16-k1-k1>`__
+ ddb-ci-ps-ll-st-t few 2017-06-28 2017-07-15 `8494 <https://bugs.opendaylight.org/show_bug.cgi?id=8494>`__ `link <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/764/log.html.gz#s1-s34-t3-k2-k16-k1-k1>`__
+ ddb-ci-ps-lr-ct-t few 2017-06-28 2017-07-15 `8494 <https://bugs.opendaylight.org/show_bug.cgi?id=8494>`__ `link <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/764/log.html.gz#s1-s34-t5-k2-k16-k1-k1>`__
+ ddb-ci-ps-lr-st-t few 2017-06-28 2017-07-15 `8494 <https://bugs.opendaylight.org/show_bug.cgi?id=8494>`__ `link <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/764/log.html.gz#s1-s34-t7-k2-k16-k1-k1>`__
+ ddb-ls-ms-lr-t tell 2017-07-15 2017-07-15 `8792 <https://bugs.opendaylight.org/show_bug.cgi?id=8792>`__ `link <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/783/log.html.gz#s1-s36-t1-k2-k12>`__
+ ddb-ls-ms-rr-t tell 2017-07-14 2017-07-15 `8792 <https://bugs.opendaylight.org/show_bug.cgi?id=8792>`__ `link <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/782/log.html.gz#s1-s36-t3-k2-k12>`__
+ ddb-ls-ms-rl-t tell 2017-07-12 2017-07-15 `8792 <https://bugs.opendaylight.org/show_bug.cgi?id=8792>`__ `link <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/779/log.html.gz#s1-s36-t5-k2-k12>`__
+ ddb-ls-ps-lr-t pass long ago 2017-07-15 `link <https://jenkins.opendaylight.org/releng/job/controller-csit-3node-clustering-only-carbon/lastSuccessfulBuild/robot/controller-clustering.txt/Listener%20Stability%20Prefbasedshard/Move_Leader_From_Listener_Local_To_Remote/>`__
+ ddb-ls-ps-rr-t few 2017-06-26 2017-07-15 `8733 <https://bugs.opendaylight.org/show_bug.cgi?id=8733>`__ `link <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/762/log.html.gz#s1-s38-t3-k2-k14-k2-k1-k4-k7-k1>`__
+ ddb-ls-ps-rl-t pass long ago 2017-07-15 `link <https://jenkins.opendaylight.org/releng/job/controller-csit-3node-clustering-only-carbon/lastSuccessfulBuild/robot/controller-clustering.txt/Listener%20Stability%20Prefbasedshard/Move_Leader_From_Listener_Remote_To_Local/>`__
+ drb-rpp-ms-a pass long ago 2017-07-15 `link <https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-3node-clustering-only-carbon/lastSuccessfulBuild/robot/controller-clustering.txt/Rpc%20Provider%20Precedence/>`__
+ drb-rph-ms-a few 2017-06-28 2017-07-15 `8430 <https://bugs.opendaylight.org/show_bug.cgi?id=8430>`__ `link <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/764/log.html.gz#s1-s4-t6-k2-k1-k1-k1-k1-k1-k1-k2-k1-k1-k1-k3-k1-k1-k1-k2-k1-k4-k7-k1>`__
+ drb-app-ms-a pass long ago 2017-07-15 `link <https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-3node-clustering-only-carbon/lastSuccessfulBuild/robot/controller-clustering.txt/Action%20Provider%20Precedence/>`__
+ drb-aph-ms-a few 2017-07-02 2017-07-15 `8430 <https://bugs.opendaylight.org/show_bug.cgi?id=8430>`__ `link <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/769/log.html.gz#s1-s8-t6-k2-k1-k1-k1-k1-k1-k1-k1-k1-k1-k1-k3-k1-k1-k1-k3-k1-k4-k7-k1>`__
+ dnb-1n-60k-a pass long ago 2017-07-15 `link <https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-1node-rest-cars-perf-only-carbon/lastSuccessfulBuild/robot/controller-rest-cars-perf.txt/Noloss%20Rate%201Node/>`__
+ ss-ms-ms-a pass long ago 2017-07-15 `link <https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-3node-clustering-only-carbon/lastSuccessfulBuild/robot/controller-clustering.txt/Master%20Stability/>`__
+ ss-ph-ms-a few 2017-06-29 2017-07-15 `8420 <https://bugs.opendaylight.org/show_bug.cgi?id=8420>`__ `link <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/765/log.html.gz#s1-s12-t5-k2-k3-k1-k2>`__
+ ss-cl-ms-a pass long ago 2017-07-15 `link <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/lastSuccessfulBuild/log.html.gz#s1-s14>`__
+ ss-ms-ms-t pass long ago 2017-07-15 `link <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/lastSuccessfulBuild/log.html.gz#s1-s40>`__
+ ss-ph-ms-t few 2017-07-15 2017-07-15 `8420 <https://bugs.opendaylight.org/show_bug.cgi?id=8420>`__ `link <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/783/log.html.gz#s1-s42-t5-k2-k3-k1-k2>`__
+ ss-cl-ms-t pass long ago 2017-07-15 `link <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/lastSuccessfulBuild/log.html.gz#s1-s44>`__
+ netconf-ba-ms-a pass long ago 2017-07-14 `link <https://jenkins.opendaylight.org/releng/view/netconf/job/netconf-csit-3node-clustering-only-carbon/lastSuccessfulBuild/robot/netconf-clustering.txt/CRUD>`__
+ netconf-ok-ms-a few 2017-06-18 2017-07-14 `8596 <https://bugs.opendaylight.org/show_bug.cgi?id=8596>`__ `link <https://logs.opendaylight.org/releng/jenkins092/netconf-csit-3node-clustering-only-carbon/568/log.html.gz#s1-s5-t17-k2-k3-k2-k2-k1>`__
+ netconf-rr-ms-a pass long ago 2017-07-14 `link <https://jenkins.opendaylight.org/releng/view/netconf/job/netconf-csit-3node-clustering-only-carbon/lastSuccessfulBuild/robot/netconf-clustering.txt/Outages>`__
+ bgp-3n-300k-t-long akka 2017-07-08 2017-07-08 `8318 <https://bugs.opendaylight.org/show_bug.cgi?id=8318>`__ `link <https://logs.opendaylight.org/releng/jenkins092/bgpcep-csit-3node-bgpclustering-longevity-only-carbon/11/log.html.gz#s1-s2-t1-k10-k1-k1-k1-k1-k1-k1-k1-k1-k1-k2-k1-k3-k7-k5-k1-k6-k1-k1-k1-k1-k1-k2-k1-k1-k2-k2-k2-k1-k6-k2-k2-k1>`__
+ ddb-elm-mc-t-long tell 2017-07-08 2017-07-08 `8618 <https://bugs.opendaylight.org/show_bug.cgi?id=8618>`__ `link <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-ddb-expl-lead-movement-longevity-only-carbon/14/log.html.gz#s1-s2-t1-k2-k1-k1-k1-k1-k1-k1-k2-k1-k1-k2-k10>`__
+ drb-rpp-ms-a-long few 2017-05-07 2017-07-08 `8430 <https://bugs.opendaylight.org/show_bug.cgi?id=8430>`__ `link <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-drb-partnheal-longevity-only-carbon/13/console.log.gz>`__
+ drb-rph-ms-a-long akka 2017-07-08 2017-07-08 `8430 <https://bugs.opendaylight.org/show_bug.cgi?id=8430>`__ `link <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-drb-partnheal-longevity-only-carbon/18/log.html.gz#s1-t1-k3-k1-k1-k1-k1-k1-k1-k2-k1-k1-k6-k1-k1-k1-k1-k1-k1-k2-k1-k1-k1-k3-k1-k1-k1-k2-k1-k4-k7-k1>`__
+ dnb-1n-60k-a-long pass long ago 2017-07-08 `link <https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-1node-rest-cars-perf-only-carbon/620/robot/controller-rest-cars-perf.txt/Noloss%20Rate%201Node/>`__
+ ss-ph-ms-a-long akka 2017-07-08 2017-07-08 `8420 <https://bugs.opendaylight.org/show_bug.cgi?id=8420>`__ `link <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-cs-partnheal-longevity-only-carbon/15/log.html.gz#s1-s2-t1-k3-k1-k1-k1-k1-k1-k1-k1-k1-k1-k7-k3-k1-k2>`__
+ ss-cl-ms-a-long pass long ago 2017-07-08 `link <https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-1node-rest-cars-perf-only-carbon/620/robot/controller-rest-cars-perf.txt/Noloss%20Rate%201Node/>`__
+ ================== ===== ========== ========== ============================================================= ==========
+
+For descriptions of test cases, see `description page <tests.html>`_.
+Note that the link contains current description,
+the details might have been implemented differently at SR1 release.
--- /dev/null
+
+Carbon SR2 test report
+^^^^^^^^^^^^^^^^^^^^^^
+
+Test Case Summary
+-----------------
+
+RelEng stability summary.
+
++ tba: Recent failures to be analyzed yet: 0.
++ test: Recent failures caused by wrong assumptions in test: 0.
++ akka: Recent failures related to pure UnreachableMember: 4.
++ tell: Recent failures not clearly caused by UnreachableMember: 4.
++ few: Tests passing unless low frequency failure happens: 7 (6 without duplication).
+ (Low frequency means infra issues or UnreachableMemeber,
+ related to Akka where Controller code has not real control.)
++ pass: Tests passing consistently: 38 (36 without duplication).
++ Total: 53 (50 without duplication).
++ Total minus akka: 49 (46 without duplication).
++ Total minus akka, passing always or mostly: 45 (42 without duplication).
++ Acceptance rate: 45/49=91.83% (42/46=91.30% without duplication).
+
+Table
+-----
+
+S017 instead of 2017 means Sandbox run (includes changes not merged to stable/carbon yet).
+
+Last fail is date of last failure not caused by infra
+(or by a typo in test or by netconf/bgp failing to initialize properly).
+
+"S 17" or "2 17" in Last run means the documented run was superseded by a newer one, but not analyzed yet.
+
+"few" status from SR1 is not inherited (such tests are marked as "pass").
+"long ago" means the last real test failue happened somewhere around SR1 release (or before that, or never).
+
+If status is a link, it points to the latest relevant robot failure, or a history to see the stability.
+In case of failure, Bugs field gives the reason of that failure.
+
+.. table:: Releng stability results (post SR1, pre SR2)
+ :widths: 40,15,15,15,15
+
+ =================== ========== ========== ============================================================= ======
+ Test case Last fail Last run Bugs Status
+ =================== ========== ========== ============================================================= ======
+ bgp-1n-300k-a_ long ago 2017-09-18 `PASS <https://jenkins.opendaylight.org/releng/view/bgpcep/job/bgpcep-csit-1node-periodic-bgp-ingest-all-carbon/lastSuccessfulBuild/robot/bgpcep-bgp-ingest.txt/Singlepeer%20Pc%20Shm%20300Kroutes/>`__
+ bgp-1n-300k-t_ long ago 2017-09-18 `PASS <https://jenkins.opendaylight.org/releng/view/bgpcep/job/bgpcep-csit-1node-periodic-bgp-ingest-all-carbon/lastSuccessfulBuild/robot/bgpcep-bgp-ingest.txt/Singlepeer%20Pc%20Shm%20300Kroutes_1/>`__
+ bgp-3n-300k-ll-t_ 2017-09-16 2017-09-18 `8318 <https://bugs.opendaylight.org/show_bug.cgi?id=8318>`__ `AKKA <https://logs.opendaylight.org/releng/jenkins092/bgpcep-csit-3node-periodic-bgpclustering-all-carbon/401/log.html.gz#s1-s2-t11-k2-k2>`__
+ bgp-3n-300k-lr-t_ 2017-09-16 2017-09-18 `8318 <https://bugs.opendaylight.org/show_bug.cgi?id=8318>`__ `AKKA <https://logs.opendaylight.org/releng/jenkins092/bgpcep-csit-3node-periodic-bgpclustering-all-carbon/401/log.html.gz#s1-s4-t8-k2-k3-k7-k4-k1-k6-k1-k1-k1-k1-k1-k2-k1-k4>`__
+ ddb-cls-ms-ll-t_ 2017-08-24 2017-09-18 `PASS <https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-3node-clustering-all-carbon/lastSuccessfulBuild/robot/controller-clustering.txt/Clean%20Leader%20Shutdown/Local_Leader_Shutdown>`__
+ ddb-cls-ms-lr-t_ long ago 2017-09-18 `PASS <https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-3node-clustering-all-carbon/lastSuccessfulBuild/robot/controller-clustering.txt/Clean%20Leader%20Shutdown/Remote_Leader_Shutdown>`__
+ ddb-cls-ps-ll-t_ long ago 2017-09-18 `PASS <https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-3node-clustering-all-carbon/lastSuccessfulBuild/robot/controller-clustering.txt/Clean%20Leader%20Shutdown%20Prefbasedshard/Local_Leader_Shutdown>`__
+ ddb-cls-ps-lr-t_ long ago 2017-09-18 `PASS <https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-3node-clustering-all-carbon/lastSuccessfulBuild/robot/controller-clustering.txt/Clean%20Leader%20Shutdown%20Prefbasedshard/Remote_Leader_Shutdown>`__
+ ddb-elm-ms-lr-t_ long ago 2017-09-18 `PASS <https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-3node-clustering-all-carbon/lastSuccessfulBuild/robot/controller-clustering.txt/Explicit%20Leader%20Movement/Local_To_Remote_Movement>`__
+ ddb-elm-ms-rr-t_ long ago 2017-09-18 `PASS <https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-3node-clustering-all-carbon/lastSuccessfulBuild/robot/controller-clustering.txt/Explicit%20Leader%20Movement/Remote_To_Remote_Movement>`__
+ ddb-elm-ms-rl-t_ long ago 2017-09-18 `PASS <https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-3node-clustering-all-carbon/lastSuccessfulBuild/robot/controller-clustering.txt/Explicit%20Leader%20Movement/Remote_To_Local_Movement>`__
+ ddb-elm-ps-lr-t_ long ago 2017-09-18 `PASS <https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-3node-clustering-all-carbon/lastSuccessfulBuild/robot/controller-clustering.txt/Explicit%20Leader%20Movement%20Prefbasedshard/Local_To_Remote_Movement>`__
+ ddb-elm-ps-rr-t_ long ago 2017-09-18 `PASS <https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-3node-clustering-all-carbon/lastSuccessfulBuild/robot/controller-clustering.txt/Explicit%20Leader%20Movement%20Prefbasedshard/Remote_To_Remote_Movement>`__
+ ddb-elm-ps-rl-t_ long ago 2017-09-18 `PASS <https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-3node-clustering-all-carbon/lastSuccessfulBuild/robot/controller-clustering.txt/Explicit%20Leader%20Movement%20Prefbasedshard/Remote_To_Local_Movement>`__
+ ddb-li-ms-st-t_ 2017-08-18 2017-09-18 `PASS <https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-3node-clustering-all-carbon/lastSuccessfulBuild/robot/controller-clustering.txt/Leader%20Isolation/Healing_Within_Request_Timeout>`__
+ ddb-li-ms-dt-t_ 2017-08-21 2017-09-18 `PASS <https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-3node-clustering-all-carbon/lastSuccessfulBuild/robot/controller-clustering.txt/Leader%20Isolation/Healing_After_Request_Timeout>`__
+ ddb-li-ps-st-t_ 2017-09-01 2017-09-18 `PASS <https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-3node-clustering-all-carbon/lastSuccessfulBuild/robot/controller-clustering.txt/Leader%20Isolation%20Prefbasedshard/Healing_Within_Request_Timeout>`__
+ ddb-li-ps-dt-t_ 2017-09-18 2017-09-18 `8845 <https://bugs.opendaylight.org/show_bug.cgi?id=8845>`__ `TELL <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-all-carbon/445/log.html.gz#s1-s30-t3-k2-k25-k1-k8>`__
+ ddb-ci-ms-ll-ct-t_ long ago 2017-09-18 `PASS <https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-3node-clustering-all-carbon/lastSuccessfulBuild/robot/controller-clustering.txt/Client%20Isolation/Producer_On_Shard_Leader_Node_ChainedTx>`__
+ ddb-ci-ms-ll-st-t_ long ago 2017-09-18 `PASS <https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-3node-clustering-all-carbon/lastSuccessfulBuild/robot/controller-clustering.txt/Client%20Isolation/Producer_On_Shard_Leader_Node_SimpleTx>`__
+ ddb-ci-ms-lr-ct-t_ long ago 2017-09-18 `PASS <https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-3node-clustering-all-carbon/lastSuccessfulBuild/robot/controller-clustering.txt/Client%20Isolation/Producer_On_Shard_Non_Leader_Node_ChainedTx>`__
+ ddb-ci-ms-lr-st-t_ long ago 2017-09-18 `PASS <https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-3node-clustering-all-carbon/lastSuccessfulBuild/robot/controller-clustering.txt/Client%20Isolation/Producer_On_Shard_Non_Leader_Node_SimpleTx>`__
+ ddb-ci-ps-ll-it-t_ long ago 2017-09-18 `PASS <https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-3node-clustering-all-carbon/lastSuccessfulBuild/robot/controller-clustering.txt/Client%20Isolation%20Prefbasedshard/Producer_On_Shard_Leader_Node_Isolated_Transactions>`__
+ ddb-ci-ps-ll-nt-t_ long ago 2017-09-18 `PASS <https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-3node-clustering-all-carbon/lastSuccessfulBuild/robot/controller-clustering.txt/Client%20Isolation%20Prefbasedshard/Producer_On_Shard_Leader_Node_Nonisolated_Transactions>`__
+ ddb-ci-ps-lr-it-t_ long ago 2017-09-18 `PASS <https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-3node-clustering-all-carbon/lastSuccessfulBuild/robot/controller-clustering.txt/Client%20Isolation%20Prefbasedshard/Producer_On_Shard_Non_Leader_Node_Isolated_Transactions>`__
+ ddb-ci-ps-lr-nt-t_ long ago 2017-09-18 `PASS <https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-3node-clustering-all-carbon/lastSuccessfulBuild/robot/controller-clustering.txt/Client%20Isolation%20Prefbasedshard/Producer_On_Shard_Non_Leader_Node_Nonisolated_Transactions>`__
+ ddb-ls-ms-lr-t_ long ago 2017-09-18 `PASS <https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-3node-clustering-all-carbon/lastSuccessfulBuild/robot/controller-clustering.txt/Listener%20Stability/Move_Leader_From_Listener_Local_To_Remote>`__
+ ddb-ls-ms-rr-t_ long ago 2017-09-18 `PASS <https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-3node-clustering-all-carbon/lastSuccessfulBuild/robot/controller-clustering.txt/Listener%20Stability/Move_Leader_From_Listener_Remote_To_Other_Remote>`__
+ ddb-ls-ms-rl-t_ long ago 2017-09-18 `PASS <https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-3node-clustering-all-carbon/lastSuccessfulBuild/robot/controller-clustering.txt/Listener%20Stability/Move_Leader_From_Listener_Remote_To_Local>`__
+ ddb-ls-ps-lr-t_ 2017-09-18 2017-09-18 `8733 <https://bugs.opendaylight.org/show_bug.cgi?id=8733>`__ `TELL <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-all-carbon/445/log.html.gz#s1-s38-t1-k2-k14-k2-k1-k4-k7-k1>`__
+ ddb-ls-ps-rr-t_ 2017-09-18 2017-09-18 `8733 <https://bugs.opendaylight.org/show_bug.cgi?id=8733>`__ `TELL <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-all-carbon/445/log.html.gz#s1-s38-t3-k2-k14-k2-k1-k4-k7-k1>`__
+ ddb-ls-ps-rl-t_ 2017-09-18 2017-09-18 `8733 <https://bugs.opendaylight.org/show_bug.cgi?id=8733>`__ `FEW <https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-3node-clustering-all-carbon/lastSuccessfulBuild/robot/controller-clustering.txt/Listener%20Stability%20Prefbasedshard/Move_Leader_From_Listener_Remote_To_Local/>`__
+ drb-rpp-ms-a_ long ago 2017-09-18 `PASS <https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-3node-clustering-all-carbon/lastSuccessfulBuild/robot/controller-clustering.txt/Rpc%20Provider%20Precedence>`__
+ drb-rph-ms-a_ long ago 2017-09-18 `PASS <https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-3node-clustering-all-carbon/lastSuccessfulBuild/robot/controller-clustering.txt/Rpc%20Provider%20Partition%20And%20Heal>`__
+ drb-app-ms-a_ long ago 2017-09-18 `PASS <https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-3node-clustering-all-carbon/lastSuccessfulBuild/robot/controller-clustering.txt/Action%20Provider%20Precedence>`__
+ drb-aph-ms-a_ long ago 2017-09-18 `PASS <https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-3node-clustering-all-carbon/lastSuccessfulBuild/robot/controller-clustering.txt/Action%20Provider%20Partition%20And%20Heal>`__
+ dnb-1n-60k-a_ long ago 2017-09-18 `PASS <https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-1node-rest-cars-perf-all-carbon/lastSuccessfulBuild/robot/controller-rest-cars-perf.txt/Noloss%20Rate%201Node/>`__
+ ss-ms-ms-a_ long ago 2017-09-18 `PASS <https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-3node-clustering-all-carbon/lastSuccessfulBuild/robot/controller-clustering.txt/Master%20Stability>`__
+ ss-ph-ms-a_ 2017-09-01 2017-09-18 `8420 <https://bugs.opendaylight.org/show_bug.cgi?id=8420>`__ `FEW <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-all-carbon/427/log.html.gz#s1-s12-t5-k2-k3-k1-k2>`__
+ ss-cl-ms-a_ long ago 2017-09-18 `PASS <https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-3node-clustering-all-carbon/lastSuccessfulBuild/robot/controller-clustering.txt/Chasing%20The%20Leader>`__
+ ss-ms-ms-t_ long ago 2017-09-18 `PASS <https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-3node-clustering-all-carbon/lastSuccessfulBuild/robot/controller-clustering.txt/Master%20Stability_1>`__
+ ss-ph-ms-t_ 2017-09-17 2017-09-18 `9177 <https://bugs.opendaylight.org/show_bug.cgi?id=9177>`__ `FEW <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-all-carbon/444/log.html.gz#s1-s42-t5-k2-k2-k1-k2-k1-k2-k1-k6-k1-k2-k1>`__
+ ss-cl-ms-t_ long ago 2017-09-18 `PASS <https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-3node-clustering-all-carbon/lastSuccessfulBuild/robot/controller-clustering.txt/Chasing%20The%20Leader_1>`__
+ netconf-ba-ms-a_ long ago 2017-09-18 `PASS <https://jenkins.opendaylight.org/releng/view/netconf/job/netconf-csit-3node-clustering-all-carbon/615/robot/netconf-clustering.txt/CRUD>`__
+ netconf-ok-ms-a_ long ago 2017-09-18 `PASS <https://jenkins.opendaylight.org/releng/view/netconf/job/netconf-csit-3node-clustering-all-carbon/lastSuccessfulBuild/robot/netconf-clustering.txt/Entity/>`__
+ netconf-rr-ms-a_ 2017-09-06 2017-09-18 `9006 <https://bugs.opendaylight.org/show_bug.cgi?id=9027>`__ `TELL <https://logs.opendaylight.org/releng/jenkins092/netconf-csit-3node-clustering-all-carbon/394/log.html.gz#s1-s9-t9-k2-k2-k8-k1-k2-k1-k1-k2-k1-k4-k1>`__
+ bgp-3n-300k-t-long_ 2017-09-16 2017-09-16 `8318 <https://bugs.opendaylight.org/show_bug.cgi?id=8318>`__ `AKKA <https://logs.opendaylight.org/releng/jenkins092/bgpcep-csit-3node-bgpclustering-longevity-only-carbon/21/log.html.gz#s1-s2-t1-k10-k1-k1-k1-k1-k1-k1-k1-k1-k1-k2-k2-k3-k7-k1-k1-k6-k1-k1-k1-k1-k1-k2-k1-k1-k2-k2-k2-k1-k6-k3-k1-k1-k5-k1-k3-k1>`__
+ ddb-elm-mc-t-long_ 2017-08-06 2017-09-16 `FEW <https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-3node-ddb-expl-lead-movement-longevity-only-carbon/lastSuccessfulBuild/robot/controller-ddb-expl-lead-movement-longevity.txt/Explicit%20Leader%20Movement%20Longevity/>`__
+ drb-rpp-ms-a-long_ long ago 2017-09-16 `FEW <https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-3node-drb-precedence-longevity-only-carbon/lastSuccessfulBuild/robot/controller-drb-precedence-longevity.txt/Rpc_Provider_Precedence_Longevity/>`__
+ drb-rph-ms-a-long_ 2017-08-12 2017-09-16 `PASS <https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-3node-drb-partnheal-longevity-only-carbon/lastSuccessfulBuild/robot/controller-drb-partnheal-longevity.txt/Rpc_Provider_Precedence_Longevity/>`__
+ dnb-1n-60k-a-long_ long ago 2017-09-16 `FEW <https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-1node-notifications-longevity-only-carbon/lastSuccessfulBuild/robot/controller-notifications-longevity.txt/Notifications_longevity/>`__
+ ss-ph-ms-a-long_ 2017-09-16 2017-09-16 `8420 <https://bugs.opendaylight.org/show_bug.cgi?id=8420>`__ `AKKA <https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-cs-partnheal-longevity-only-carbon/26/log.html.gz#s1-s2-t1-k3-k1-k1-k4>`__
+ ss-cl-ms-a-long_ 2017-08-06 2017-09-16 `PASS <https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-3node-cs-chasing-leader-longevity-only-carbon/lastSuccessfulBuild/robot/controller-cs-chasing-leader-longevity.txt/Chasing%20The%20Leader%20Longevity/>`__
+ =================== ========== ========== ============================================================= ======
+
+.. _bgp-1n-300k-a: tests.html#bgp-1n-300k-a
+.. _bgp-1n-300k-t: tests.html#bgp-1n-300k-t
+.. _bgp-3n-300k-ll-t: tests.html#bgp-3n-300k-ll-t
+.. _bgp-3n-300k-lr-t: tests.html#bgp-3n-300k-lr-t
+.. _ddb-cls-ms-ll-t: tests.html#ddb-cls-ms-ll-t
+.. _ddb-cls-ms-lr-t: tests.html#ddb-cls-ms-lr-t
+.. _ddb-cls-ps-ll-t: tests.html#ddb-cls-ps-ll-t
+.. _ddb-cls-ps-lr-t: tests.html#ddb-cls-ps-lr-t
+.. _ddb-elm-ms-lr-t: tests.html#ddb-elm-ms-lr-t
+.. _ddb-elm-ms-rr-t: tests.html#ddb-elm-ms-rr-t
+.. _ddb-elm-ms-rl-t: tests.html#ddb-elm-ms-rl-t
+.. _ddb-elm-ps-lr-t: tests.html#ddb-elm-ps-lr-t
+.. _ddb-elm-ps-rr-t: tests.html#ddb-elm-ps-rr-t
+.. _ddb-elm-ps-rl-t: tests.html#ddb-elm-ps-rl-t
+.. _ddb-li-ms-st-t: tests.html#ddb-li-ms-st-t
+.. _ddb-li-ms-dt-t: tests.html#ddb-li-ms-dt-t
+.. _ddb-li-ps-st-t: tests.html#ddb-li-ps-st-t
+.. _ddb-li-ps-dt-t: tests.html#ddb-li-ps-dt-t
+.. _ddb-ci-ms-ll-ct-t: tests.html#ddb-ci-ms-ll-ct-t
+.. _ddb-ci-ms-ll-st-t: tests.html#ddb-ci-ms-ll-st-t
+.. _ddb-ci-ms-lr-ct-t: tests.html#ddb-ci-ms-lr-ct-t
+.. _ddb-ci-ms-lr-st-t: tests.html#ddb-ci-ms-lr-st-t
+.. _ddb-ci-ps-ll-it-t: tests.html#ddb-ci-ps-ll-it-t
+.. _ddb-ci-ps-ll-nt-t: tests.html#ddb-ci-ps-ll-nt-t
+.. _ddb-ci-ps-lr-it-t: tests.html#ddb-ci-ps-lr-it-t
+.. _ddb-ci-ps-lr-nt-t: tests.html#ddb-ci-ps-lr-nt-t
+.. _ddb-ls-ms-lr-t: tests.html#ddb-ls-ms-lr-t
+.. _ddb-ls-ms-rr-t: tests.html#ddb-ls-ms-rr-t
+.. _ddb-ls-ms-rl-t: tests.html#ddb-ls-ms-rl-t
+.. _ddb-ls-ps-lr-t: tests.html#ddb-ls-ps-lr-t
+.. _ddb-ls-ps-rr-t: tests.html#ddb-ls-ps-rr-t
+.. _ddb-ls-ps-rl-t: tests.html#ddb-ls-ps-rl-t
+.. _drb-rpp-ms-a: tests.html#drb-rpp-ms-a
+.. _drb-rph-ms-a: tests.html#drb-rph-ms-a
+.. _drb-app-ms-a: tests.html#drb-app-ms-a
+.. _drb-aph-ms-a: tests.html#drb-aph-ms-a
+.. _dnb-1n-60k-a: tests.html#dnb-1n-60k-a
+.. _ss-ms-ms-a: tests.html#ss-ms-ms-a
+.. _ss-ph-ms-a: tests.html#ss-ph-ms-a
+.. _ss-cl-ms-a: tests.html#ss-cl-ms-a
+.. _ss-ms-ms-t: tests.html#ss-ms-ms-t
+.. _ss-ph-ms-t: tests.html#ss-ph-ms-t
+.. _ss-cl-ms-t: tests.html#ss-cl-ms-t
+.. _netconf-ba-ms-a: tests.html#netconf-ba-ms-a
+.. _netconf-ok-ms-a: tests.html#netconf-ok-ms-a
+.. _netconf-rr-ms-a: tests.html#netconf-rr-ms-a
+.. _bgp-3n-300k-t-long: tests.html#bgp-3n-300k-t-long
+.. _ddb-elm-mc-t-long: tests.html#ddb-elm-mc-t-long
+.. _drb-rpp-ms-a-long: tests.html#drb-rpp-ms-a-long
+.. _drb-rph-ms-a-long: tests.html#drb-rph-ms-a-long
+.. _dnb-1n-60k-a-long: tests.html#dnb-1n-60k-a-long
+.. _ss-ph-ms-a-long: tests.html#ss-ph-ms-a-long
+.. _ss-cl-ms-a-long: tests.html#ss-cl-ms-a-long
--- /dev/null
+
+List of test cases
+^^^^^^^^^^^^^^^^^^
+
+Each test case has a shorter code, tables with results use that code.
+In result tables, the code is a link to this document,
+due to coala ReST requirements, the codes are (self-pointing) links also in this document.
+
+Other links point to scenario definitions ao caveat items.
+
++ `DOMDataBroker`_: Producers make 1000 transactions per second, except BGP which works full speed.
+
+ + `Leader stability`_: BGP inject benchmark (thus module shards only), `300k prefixes`_, 1 Python peer. Progress tracked by counting prefixes in example-ipv4-topology.
+
+ + Ask-based protocol:
+
+ .. _bgp-1n-300k-a:
+
+ + Single member: bgp-1n-300k-a_
+
+ + `Tell-based protocol`_:
+
+ .. _bgp-1n-300k-t:
+
+ + Single member: bgp-1n-300k-t_
+
+ + Three members:
+
+ .. _bgp-3n-300k-ll-t:
+
+ + Leaders local: bgp-3n-300k-ll-t_
+
+ .. _bgp-3n-300k-lr-t:
+
+ + `Leaders remote`_: bgp-3n-300k-lr-t_
+
+ .. _bgp-3n-300k-t-long:
+
+ + `Longevity`_: bgp-3n-300k-t-long_
+
+ + `Clean leader shutdown`_, `Tell-based protocol`_:
+
+ + Module-based shards:
+
+ .. _ddb-cls-ms-ll-t:
+
+ + Shard leader local to producer: ddb-cls-ms-ll-t_
+
+ .. _ddb-cls-ms-lr-t:
+
+ + Shard leader remote to producer: ddb-cls-ms-lr-t_
+
+ + `Prefix-based shards`_:
+
+ .. _ddb-cls-ps-ll-t:
+
+ + Shard leader local to producer: ddb-cls-ps-ll-t_
+
+ .. _ddb-cls-ps-lr-t:
+
+ + Shard leader remote to producer: ddb-cls-ps-lr-t_
+
+ + `Explicit leader movement`_, `Tell-based protocol`_:
+
+ + Module-based shards:
+
+ .. _ddb-elm-ms-lr-t:
+
+ + Local leader to remote: ddb-elm-ms-lr-t_
+
+ .. _ddb-elm-ms-rr-t:
+
+ + Remote leader to other remote: ddb-elm-ms-rr-t_
+
+ .. _ddb-elm-ms-rl-t:
+
+ + Remote leader to local: ddb-elm-ms-rl-t_
+
+ .. _ddb-elm-mc-t-long:
+
+ + Longevity (randomized direction): ddb-elm-mc-t-long_
+
+ + `Prefix-based shards`_:
+
+ .. _ddb-elm-ps-lr-t:
+
+ + Local leader to remote: ddb-elm-ps-lr-t_
+
+ .. _ddb-elm-ps-rr-t:
+
+ + Remote leader to other remote: ddb-elm-ps-rr-t_
+
+ .. _ddb-elm-ps-rl-t:
+
+ + Remote leader to local: ddb-elm-ps-rl-t_
+
+ + `Leader isolation`_ (`network partition only`_), `Tell-based protocol`_:
+
+ + Module-based shards:
+
+ .. _ddb-li-ms-st-t:
+
+ + Heal within transaction timeout: ddb-li-ms-st-t_
+
+ .. _ddb-li-ms-dt-t:
+
+ + Heal after transaction timeout: ddb-li-ms-dt-t_
+
+ + `Prefix-based shards`_:
+
+ .. _ddb-li-ps-st-t:
+
+ + Heal within transaction timeout: ddb-li-ps-st-t_
+
+ .. _ddb-li-ps-dt-t:
+
+ + Heal after transaction timeout: ddb-li-ps-dt-t_
+
+ + `Client isolation`_, `Tell-based protocol`_:
+
+ + Module-based shards:
+
+ + Leader local:
+
+ .. _ddb-ci-ms-ll-st-t:
+
+ + `Simple transactions`_: ddb-ci-ms-ll-st-t_
+
+ .. _ddb-ci-ms-ll-ct-t:
+
+ + Transaction chain: ddb-ci-ms-ll-ct-t_
+
+ + Leader remote:
+
+ .. _ddb-ci-ms-lr-st-t:
+
+ + Simple transactions: ddb-ci-ms-lr-st-t_
+
+ .. _ddb-ci-ms-lr-ct-t:
+
+ + Transaction chain: ddb-ci-ms-lr-ct-t_
+
+ + `Prefix-based shards`_:
+
+ + Leader local:
+
+ .. _ddb-ci-ps-ll-it-t:
+
+ + Isolated transactions: ddb-ci-ps-ll-it-t_
+
+ .. _ddb-ci-ps-ll-nt-t:
+
+ + Non-isolated transactions: ddb-ci-ps-ll-nt-t_
+
+ + Leader remote:
+
+ .. _ddb-ci-ps-lr-it-t:
+
+ + Isolated transactions: ddb-ci-ps-lr-it-t_
+
+ .. _ddb-ci-ps-lr-nt-t:
+
+ + Non-isolated transactions: ddb-ci-ps-lr-nt-t_
+
+ + `Listener stablity`_, `Tell-based protocol`_:
+
+ + Module-based shards:
+
+ .. _ddb-ls-ms-lr-t:
+
+ + Local to remote: ddb-ls-ms-lr-t_
+
+ .. _ddb-ls-ms-rr-t:
+
+ + Remote to remote: ddb-ls-ms-rr-t_
+
+ .. _ddb-ls-ms-rl-t:
+
+ + Remote to local: ddb-ls-ms-rl-t_
+
+ + `Prefix-based shards`_:
+
+ .. _ddb-ls-ps-lr-t:
+
+ + Local to remote: ddb-ls-ps-lr-t_
+
+ .. _ddb-ls-ps-rr-t:
+
+ + Remote to remote: ddb-ls-ps-rr-t_
+
+ .. _ddb-ls-ps-rl-t:
+
+ + Remote to local: ddb-ls-ps-rl-t_
+
++ `DOMRpcBroker`_, ask-based protocol:
+
+ + `RPC Provider Precedence`_:
+
+ .. _drb-rpp-ms-a:
+
+ + Functional: drb-rpp-ms-a_
+
+ .. _drb-rpp-ms-a-long:
+
+ + Longevity: drb-rpp-ms-a-long_
+
+ + `RPC Provider Partition and Heal`_:
+
+ .. _drb-rph-ms-a:
+
+ + Functional: drb-rph-ms-a_
+
+ .. _drb-rph-ms-a-long:
+
+ + Longevity: drb-rph-ms-a-long_
+
+ .. _drb-app-ms-a:
+
+ + `Action Provider Precedence`_: drb-app-ms-a_
+
+ .. _drb-aph-ms-a:
+
+ + `Action Provider Partition and Heal`_: drb-aph-ms-a_
+
++ `DOMNotificationBroker`_: Only for 1 member, ask-based protocol.
+
+ + `No-loss rate`_: Publisher-subscriber pairs, 5k nps per pair.
+
+ .. _dnb-1n-60k-a:
+
+ + Functional (5 minute tests for 1, 4 and 12 pairs): dnb-1n-60k-a_
+
+ .. _dnb-1n-60k-a-long:
+
+ + Longevity (12 pairs): dnb-1n-60k-a-long_
+
++ `Cluster Singleton`_:
+
+ + Ask-based protocol:
+
+ .. _ss-ms-ms-a:
+
+ + `Master Stability`_: ss-ms-ms-a_
+
+ + `Partition and Heal`_:
+
+ .. _ss-ph-ms-a:
+
+ + Functional: ss-ph-ms-a_
+
+ .. _ss-ph-ms-a-long:
+
+ + Longevity: ss-ph-ms-a-long_
+
+ + `Chasing the Leader`_:
+
+ .. _ss-cl-ms-a:
+
+ + Functional: ss-cl-ms-a_
+
+ .. _ss-cl-ms-a-long:
+
+ + Longevity: ss-cl-ms-a-long_
+
+ + `Tell-based protocol`_:
+
+ .. _ss-ms-ms-t:
+
+ + `Master Stability`_: ss-ms-ms-t_
+
+ .. _ss-ph-ms-t:
+
+ + `Partition and Heal`_: ss-ph-ms-t_
+
+ .. _ss-cl-ms-t:
+
+ + `Chasing the Leader`_: ss-cl-ms-t_
+
++ `Netconf system tests`_ (ask-based protocol, module-based shards):
+
+ .. _netconf-ba-ms-a:
+
+ + `Basic access`_: netconf-ba-ms-a_
+
+ .. _netconf-ok-ms-a:
+
+ + `Owner killed`_: netconf-ok-ms-a_
+
+ .. _netconf-rr-ms-a:
+
+ + `Rolling restarts`_: netconf-rr-ms-a_
+
+.. _`300k prefixes`: caveats.html#reduced-bgp-scaling
+.. _`Action Provider Partition and Heal`: scenarios.html#action-provider-partition-and-heal
+.. _`Action Provider Precedence`: scenarios.html#action-provider-precedence
+.. _`Basic access`: scenarios.html#basic-configuration-and-mount-point-access
+.. _`Chasing the Leader`: scenarios.html#chasing-the-leader
+.. _`Clean leader shutdown`: scenarios.html#clean-leader-shutdown
+.. _`Client isolation`: scenarios.html#client-isolation
+.. _`Cluster Singleton`: scenarios.html#cluster-singleton
+.. _`DOMDataBroker`: scenarios.html#domdatabroker
+.. _`DOMNotificationBroker`: scenarios.html#domnotificationbroker
+.. _`DOMRpcBroker`: scenarios.html#domrpcbroker
+.. _`Explicit leader movement`: scenarios.html#explicit-leader-movement
+.. _`Leader isolation`: scenarios.html#leader-isolation
+.. _`Leader stability`: scenarios.html#leader-stability
+.. _`Leaders remote`: caveats.html#initial-leader-placement
+.. _`Listener stablity`: scenarios.html#listener-isolation
+.. _`Longevity`: scenarios.html#controller-cluster-services-longevity-tests
+.. _`Master Stability`: scenarios.html#master-stability
+.. _`Netconf system tests`: scenarios.html#netconf-system-tests
+.. _`network partition only`: caveats.html#isolation-mechanics
+.. _`No-loss rate`: scenarios.html#no-loss-rate
+.. _`Owner killed`: scenarios.html#device-owner-killed
+.. _`Partition and Heal`: scenarios.html#partition-and-heal
+.. _`Prefix-based shards`: caveats.html#prefix-based-shards
+.. _`Rolling restarts`: scenarios.html#rolling-restarts
+.. _`RPC Provider Partition and Heal`: scenarios.html#rpc-provider-partition-and-heal
+.. _`RPC Provider Precedence`: scenarios.html#rpc-provider-precedence
+.. _`Simple transactions`: caveats.html#producer-options
+.. _`Tell-based protocol`: caveats.html#tell-based-protocol
--- /dev/null
+Cluster testing
+===============
+
+Contents:
+
+.. toctree::
+ :maxdepth: 1
+
+ carbon/index
# General information about the project.
project = 'Integration/Test'
-copyright = '2016, Luis Gomez, Jamo Luhrsen, Daniel Farrell, Vratko Polak, Venkatrangan Govindarajan'
+copyright = '2017, Luis Gomez, Jamo Luhrsen, Daniel Farrell, Vratko Polak, Venkatrangan Govindarajan'
author = 'Luis Gomez, Jamo Luhrsen, Daniel Farrell, Vratko Polak, Venkatrangan Govindarajan'
# The version info for the project you're documenting, acts as replacement for
Contents:
.. toctree::
- :maxdepth: 2
+ :maxdepth: 1
+
+ cluster/index
External resources:
Code Review / integration / test.git / commitdiff