[integration/test.git] / docs / cluster / carbon / caveats.rst

=======
Caveats
=======

This sub-page describes ways the test implementation (or results) differs
from the `original specification <scenarios.html>`_ and which information motivates the difference.

Jenkins job structure
~~~~~~~~~~~~~~~~~~~~~

+ Information

At the start of test implementation, all the Controller 3node test cases were added into an existing Jenkins job.

During test development it was become clear, that adding all possible tests would make the job to run too long.

Dividing the job into several smaller ones is possible, but most likely the history would be lost,
unless Linux Foundation admins figure out a way to create multiple job clones with history copied.

+ Testing consequence

Even with number of test cases reduced (see below), the job duration is around three and half hours.

+ How to fix

After Carbon SR2 release, the jobs can be split, as there will be enough time
to generate new history till Carbon SR3.

Akka bugs
~~~~~~~~~

These are bugs which need either a fix in Akka codebase,
or a workaround which would be too time-consuming to implement in ODL.

Both bugs manifest as UnreachableMember event (without intentional isolation).

Slow heartbeats
---------------

+ Information

Akka sends periodic heartbeats in order to detect when the other member is being unresponsive.

The heartbeats are being serialized into the same TCP channel as ordinary data,
which means if ODL is processing big amount of data, the heartbeats can spend a long time
in TCP (or other) buffers before being processed. When this time exceeds a specific value
(currently 6 seconds), the peer memeber is declared unreachable, generally leading to leader movement.

This affects BGP test results on 3node setup, as ODL is processing BGP data as quickly as possible,
but the current BGP implementation does not handle rib owner movement gracefully (and leader movement
is explicitly checked by the test, as the scenario dictates it should not happen).
This does not affect other data broker tests, 1000 transactions per second do not generate critical throughput.

+ Testing consequence

Three test cases are failing due to `Bug 8318 <https://bugs.opendaylight.org/show_bug.cgi?id=8318>`__.

+ How to fix

Possibly, a different akka configuration could be applied to separate akka cluster status messages
into a different TCP stream than ordinary data stream.

Otherwise, a contribution to Akka project would be needed.

Reachability gossip
-------------------

+ Information

Akka uses a gossip protocol to advertize one member's reachability to other members.
There is a logic which allows for faster detection of unreachable members,
when a member can declare its peer unreachable if it got information from another peer
which is considered more up-to-date.

Ocassionally, this logic results in undesired behavior. This is when the supposedly up-to-date peer
has been isolated and now it is rejoining. Depending on timing, this can introduce additional leader movement,
or a very brief moment when a member "forgets" RPC registrations from other member.

This is causing bugs `8420 <https://bugs.opendaylight.org/show_bug.cgi?id=8420>`__
and `8430 <https://bugs.opendaylight.org/show_bug.cgi?id=8430>`__.

+ Testing consequence

This affects "partition and heal" scenarios in singleton testing.
In functional tests, the failures are infrequent enough to consider the test mostly stable overall,
but the corresponding longevity jobs are failing consistently.

The tests for "partition and heal" scenarios in RPC testing have been changed
to tolerate wrong RPC results for 10 seconds to work around this Akka bug.

+ How to fix

This does not seem fixable on ODL level, contribution to Akka project is needed.

Missing features
~~~~~~~~~~~~~~~~

Cluster yang notifications
--------------------------

+ Information

Yang notifications are not delivered to peer members.
`Bug 2139 <https://bugs.opendaylight.org/show_bug.cgi?id=2139>`__
is only fixed for data change notifications, not Yang notifications.

`Bug 2140 <https://bugs.opendaylight.org/show_bug.cgi?id=2140>`__ tracks adding this missing functionality.

+ Testing consequence

Notification suites are running on 1-node setup only.

+ How to fix

After the funtionality is added, it will be straightforward to add 3node tests.

New features
~~~~~~~~~~~~

Tell-based protocol
-------------------

+ Information

Tell-based protocol is an alternative to ask-based protocol from Boron.
Which protocol to use is decided by a line in a configuration file
(org.opendaylight.controller.cluster.datastore.cfg).

Some scenarios are expected to fail due to known limitations of ask-based protocol.
More specifically, if a shard leader moves while a transaction is open in ask-based protocol,
the transaction will fail (AskTimeoutException).

This affects only data broker tests, not RPC calls.

+ Testing consequence

In principle, this doubles the number of configurations to be tested, but see below.

+ How to fix

It is planned for tell-based protocol to become the default setting after Carbon SR2.
After that, tests for ask-based protocol can be converted or removed.

Prefix-based shards
-------------------

+ Information

Tell-based shards are an alternative to module-based shards from Boron.
Tell-based shards can be only created dynamically (as opposed to being read from a configuration file at startup).
It is possible to use both types of shards, but data writes and reads use different API,
so any Mdsal application needs to know which API to use.

The implementation of prefix-based shards is hardwired to tell-based protocol
(even if ask-based protocol is configured as the default).

+ Testing consequence

This doubles the number of configurations to be tested, for tests related to data droker (RPCs are unaffected).

+ How to fix

ODL contains great many applications which use APIs for module-based shards.
It is expected that multiple releases would still need both types of tests cases.
Module-based shards will be deprecated and removed eventually.

Producer options
----------------

+ Information

Data producers for module-based shards can produce either chained transactions or standalone transactions.
Data producers for prefix-based shards can produce either non-isolated transactions (change notifications
can combine several transactions together) or isolated transactions.

+ Testing consequence

In principle, this results in multiple Robot test cases for the same documented scenario case, but see below.

+ How to fix

All test cases will be needed in forseeable future.
Instead, more negative test cases may need be added to verify different options lead to different behavior.

Initial leader placement
~~~~~~~~~~~~~~~~~~~~~~~~

+ Information

Some scenarios do not specify initial locations of relevant shard leaders.
Test results can depend on it in presence of bugs.

This is mostly relevant to BGP test, which has three relevant members:
Rib owner, default operation shard leader and topology operational shard leader.

+ Testing consequence

Two test cases are tested. The two shard leaders are always together, rib owner is either co-located or not.
This is done by suite moving shard leaders after detecting rib owner location.

+ How to fix

Even more placements can be tested when job duration stops being the limiting factor.

Reduced BGP scaling
~~~~~~~~~~~~~~~~~~~

+ Information

Rib owner maintains de-duplicated data structures.
Other members get serialized copies and they do not de-duplicate.

Even single node strugless to fit into 6GB heap with tell-based protocol,
see `Bug 8649 <https://bugs.opendaylight.org/show_bug.cgi?id=8649>`__.

+ Testing consequence

Scale from reported tests reduced from 1 million prefixes to 300 thousand prefixes.

+ How to fix

Other members should be able to perform de-duplication, but developing that takes effort.

In the meantime, Linux Foundation could be convinced to allow for bigger VMs,
currently limited by infrastructure available.

Increased timeouts
~~~~~~~~~~~~~~~~~~

RequestTimeoutException
-----------------------

+ Information

With tell-based protocol, restconf requests might stay open up to 120 seconds before returning an error.
Even shard state reads using Jolokia can take long time if the shard actor is busy processing other messages.

+ Testing consequence

This increases duration for tests which need to verify transaction errors do happen
after sufficiently long isolation. Also, duration is increased if a test fails on a read which is otherwise quick.

+ How to fix

This involves a trade-off between stability and responsiveness.
As MD-SAL applications rarely tolerate transaction failures, users would prefer stability.
That means relatively longer timeouts are there to stay, which means test case duration
will stay high in negative (or failing positive) tests.

Client abort timeout
--------------------

+ Information

Client abort timeout is currently set to 15 minutes. The operational consequence is
just an inability to start another data producer on a member isolated for that long.
This test has too long duration compared to its usefulness.

+ Testing consequence

This test case has never been implemented.

Instead a test with isolation shorter than 120 seconds is implemented,
the test verifies the data producer continues its operation without RequestTimeoutException.

+ How to fix

It is straighforward to add the missing test cases when job duration stops being a limiting factor.

No shard shutdown
~~~~~~~~~~~~~~~~~

+ Common information.

There are multiple RPCs offering different "severity" of shard shutdown.
For technical details see comments on `change 58580 <https://git.opendaylight.org/gerrit/58580>`__.

If tests perform rigorous teardown, the shard replica should be re-activated,
which is an operation not every RPC supports.

Listener stability suite
------------------------

+ Information

Current implementation of data listeners relies on a shard replica to be active on a member
which is to receive the notification. Until that is imroved,
`Bug 8629 <https://bugs.opendaylight.org/show_bug.cgi?id=8629>`__ prevents this scenario
from being tested as described.

+ Testing consequence

The suite uses become-leader RPC instead. This has an added benefit of test case being able to pick which member
is to become the new leader (adding one more test case when the old leader was not co-located with the listener).

Also, no teardown step is needed, the final cluster state is not missing any shard replica.

+ How to fix

The original test can be implemented when listener implementation changes.
But the test which uses become-leader might be better overall.

Clean leader shutdown suite
---------------------------

+ Information

Some implementations of shutdown RPCs have a side effect of also shutting down shard state notifier.
For details see `Bug 8794 <https://bugs.opendaylight.org/show_bug.cgi?id=8794>`__.

The remove-shard-replica RPC does not have this downside, but it changes shard configuration,
which was not intended by the original scenario definition.

+ Testing consequence

Test cases for this scenario were switched to use remove-shard-replica.

+ How to fix

There is an open debate on whether "shard shutdown" RPC with less operations (compared to remove-shard-replica)
is something user wants and should be given access to.

If yes, tests can be switched to such an RPC, assuming the shard notifier issue is also fixed.

Hard reboots between test cases
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

+ Information

Timing errors in Robot code lead to Robot being unable to restore original state without restarts.

During development, we started without any hard reboots, and that was finding bugs in teardown steps of scenarios.
But test independence was more important at that time, so current tests are less sensitive to teardown failures.

+ Testing consequence

Around 115 second per ODL reboot, this time is added to every test case running time.
Together with increased timeouts, this motivates leaving out some test cases to allow faster change verification.

+ How to fix

Ideally, we would want both jobs with hard resets and jobs without them.
The jobs without resets can be added gradually after splitting the current single job.

Isolation mechanics
~~~~~~~~~~~~~~~~~~~

+ Information

During development, it was found that freeze and kill mechanics affect the co-located java test driver
without exposing any new bugs.

Turns out AAA functionality attempts to read from datastore, so isolated member returns http status code 401.

+ Testing consequence

Only iptables filtering is used in order to reduce test job duration.

Isolated members are never queried directly. A leader member is considered isolated
when other members elect a lew leader. A member is considered rejoined
when it responds reporting itself as a follower.

+ How to fix

It is straightforward to add test cases for kill and freeze where appropriate,
but once again this can be done gradually when job duration is not a limiting factor.

Reduced number of combinations
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

+ Information

Prefix-based shards always use tell-based protocol, so suites which test them
with ask-based protocol configuration can be skipped.

Ask-based protocol is known to fail on AskTimeoutException on leader movement,
so suites which produce transactions constantly can be skipped.

Most test cases are not sensitive to data producer options.

+ Testing consequence

BGP tests and singleton tests use module-based shards only, both protocols.
Other suites related to data broker are testing only tell-based protocol, both shard types.
Netconf tests and RPC tests use module-based shards with ask-based protocol only.
Only client isolaton suite tests different producer options.

+ How to fix

More ests can be added gradually (see above).

Possibly, not every combination is worth the duration it takes,
but that could be alleviated if Linux Foundation infrastructure grows in size significantly.

Reduced performance
~~~~~~~~~~~~~~~~~~~

+ Information

In order to reduce test job duration, suites wait for minimal functionality (jolokia reporting shards are in sync)
after restarting ODL. That means unrelated karaf features might still being installed
whet test is in progress. This should not affect functional tests, but it can reduce performance observed.

The only suite observing strong enough performance inpact is `chasing the leader`_.

+ Testing consequence

Functional tests for `chasing the leader`_ suite tolerate frequencies higher than 50 un-registrations per second.
Longevity suite still requires full 100 unregistrations per second.

+ How to fix

Suite can wait for better symptom of ODL being ready, for example by requiring CPU usage to become less
that a chosen threshold.

Missing logs
~~~~~~~~~~~~

+ Information

Robot VM has only 2GB of RAM and longevity jobs tend to produce large output.xml files.

Ocasionally, a job can create karaf.log files so large they fail to download,
in extreme cases filling ODL VM disk and causing failures.

This affects mostly longevity jobs (and runs with verbose logging) if they pass.

+ Testing consequence

Robot data stored is reduced to avoid this issue, sometimes leading to less details available.
This issue is still not fully resolved, so ocassionally Robot log or karaf log is still missing
if the job in question fails in an unexpected way.

+ How to fix

It is possible for Robot test to put additional data into separate files.
Unnecessarily verbose logs could be fixed where needed.

As this limitation only hurts in newly occuring bugs, it is not really possible to entirely avoid this.

Weekend outages
~~~~~~~~~~~~~~~

+ Information

Linux foundation ifrastructure teem occasionally needs to perform changes which affect running jobs.
To reduce this impact, such changes are usually done over weekend.

Cluster testing currently contains seve longevity jobs which block resources for 23 hours.
As that is a significant portion of available resources, the longevity jobs are only run on weekend
where the impact on frequency of other job is less critical.

+ Testing consequence

Sometimes, the longevity jobs are affected by infrastructure team activities,
leading to lost results or spurious failures.
One such symptom is tracked as `Bug 8959 <https://bugs.opendaylight.org/show_bug.cgi?id=8959>`__.

+ How to fix

It might be possible to spread longevity jobs over work days. As distributing jobs manually
is not a scalable option, a considerable work would be needed to create an automatic way.

Infrastructure changes are not very frequent, and having jobs run at the same predictable time
is convenient from reporting point of view, so perhaps it is okay to keep the current setup.

.. _`chasing the leader`: scenarios.html#chasing-the-leader