Leader should always apply modifications as local 76/82576/24
authorTomas Cere <tomas.cere@pantheon.tech>
Tue, 18 Jun 2019 12:54:56 +0000 (14:54 +0200)
committerRobert Varga <robert.varga@pantheon.tech>
Sun, 19 Jan 2020 13:27:25 +0000 (14:27 +0100)
commit9b319f491af1c65705b69e8a182aab5006a2f959
treebe405ef6e141410f73cf3e7e783bac3e168fbba1
parentce805eec9919fd724ffc6ac40975f0c3314ba9d8
Leader should always apply modifications as local

Normally an entry application is as follows:
1. leader sends an append entry off to persistence and replicates it to
   followers
2. leaders creates its ClientRequestTracker
3. when the entry is done with persistence and replication leader moves its
   commit index
4. part of moving the commit index is sending an ApplyState message which
   finalizes the entry application in the DataTree
5. The ApplyState determines if a ClientRequestTracker is present and adds
   an identifier to the ApplyState message if it is. This determines the way
   in which the finalize of the entry application happens in the DataTree.
   If it is present the entry is applied as if it originated on the leader,
   if it is not present it is applied as if the node is a follower.

The problem is when the leader flaps in a leader -> follower -> leader
transition after 2. and before 4..

This would mean that the new leader no longer has the ClientRequestTracker
which was created in the previous leader state, which means that when it
starts with 5. it will create the ApplyState without an identifier
and the entry finishes up the application as if the node is a follower.

This means that it will be applied without finishCommit which means that
the transaction will be forever stuck in COMMIT_PENDING state until
the node would be restarted.

Change this up, so that the leader will apply modifications as local, even
when it looses its ClientRequestTracker and add Identifiable to payloads
which require it.

Since this code path should never occur when we are candidate, catch this
transition. As ClientRequestTracker becomes an optional entity, we hide that
as well.

JIRA: CONTROLLER-1927
Change-Id: I636f998cd62ec82ef02193261624e4a51275fb86
Signed-off-by: Tomas Cere <tomas.cere@pantheon.tech>
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
16 files changed:
opendaylight/md-sal/sal-akka-raft/src/main/java/org/opendaylight/controller/cluster/raft/behaviors/AbstractLeader.java
opendaylight/md-sal/sal-akka-raft/src/main/java/org/opendaylight/controller/cluster/raft/behaviors/AbstractRaftActorBehavior.java
opendaylight/md-sal/sal-akka-raft/src/main/java/org/opendaylight/controller/cluster/raft/behaviors/Candidate.java
opendaylight/md-sal/sal-akka-raft/src/main/java/org/opendaylight/controller/cluster/raft/behaviors/Follower.java
opendaylight/md-sal/sal-clustering-commons/src/main/java/org/opendaylight/controller/cluster/raft/protobuff/client/messages/IdentifiablePayload.java [new file with mode: 0644]
opendaylight/md-sal/sal-clustering-commons/src/main/java/org/opendaylight/controller/cluster/raft/protobuff/client/messages/Payload.java
opendaylight/md-sal/sal-distributed-datastore/src/main/java/org/opendaylight/controller/cluster/datastore/persisted/AbstractIdentifiablePayload.java
opendaylight/md-sal/sal-distributed-datastore/src/main/java/org/opendaylight/controller/cluster/datastore/persisted/CommitTransactionPayload.java
opendaylight/md-sal/sal-distributed-datastore/src/main/java/org/opendaylight/controller/cluster/datastore/shardmanager/ShardInformation.java
opendaylight/md-sal/sal-distributed-datastore/src/main/java/org/opendaylight/controller/cluster/datastore/shardmanager/ShardManager.java
opendaylight/md-sal/sal-distributed-datastore/src/test/java/org/opendaylight/controller/cluster/databroker/TestClientBackedDataStore.java
opendaylight/md-sal/sal-distributed-datastore/src/test/java/org/opendaylight/controller/cluster/datastore/DistributedDataStoreRemotingIntegrationTest.java
opendaylight/md-sal/sal-distributed-datastore/src/test/java/org/opendaylight/controller/cluster/datastore/LocalShardStore.java [new file with mode: 0644]
opendaylight/md-sal/sal-distributed-datastore/src/test/java/org/opendaylight/controller/cluster/datastore/TestDistributedDataStore.java
opendaylight/md-sal/sal-distributed-datastore/src/test/java/org/opendaylight/controller/cluster/datastore/TestShard.java
opendaylight/md-sal/sal-distributed-datastore/src/test/java/org/opendaylight/controller/cluster/datastore/shardmanager/TestShardManager.java