From: Tom Pantelis Date: Sun, 4 Sep 2016 01:08:41 +0000 (-0400) Subject: Bug 6540: EOS - Rework behavior of onPeerDown X-Git-Tag: release/boron~2 X-Git-Url: https://git.opendaylight.org/gerrit/gitweb?p=controller.git;a=commitdiff_plain;h=834f164ee449e9c5b2fd131e0196e58642e29a89;hp=834f164ee449e9c5b2fd131e0196e58642e29a89 Bug 6540: EOS - Rework behavior of onPeerDown https://git.opendaylight.org/gerrit/#/c/26808/ modified the behavior of onPeerDown to remove all the down node's candidates. However this behavior is problematic in the case when the shard leader is isolated. The majority partition will elect a new leader which temporarily results in split-brain and 2 leaders which independently attempt to remove the other side's candidates. When the partition is healed, all hell breaks loose trying to reconcile their differences. This is compounded with the singleton service because it uses 2 entities that are related to one another. To alleviate this, I reverted back to the behavior of selecting a new owner for the entities owned by the down node and leaving the down node as a candidate. In the case where the down node is the only candidate, it leaves it as the owner. This doesn't hurt anything and avoids complications with having to re-instate the down node as owner when it re-joins if it was actually isolated. The idea here is to keep its candidacy to minimize disruption until proven otherwise since we don't know if the downed node's process is actually still alive. If another node registers a candidate it will replace the down node as the owner. To handle the case where the down node actually restarted, after startup when it first hears from the leader, it sends a RemoveAllCandidates message to the leader to remove it from all entities. This cleans out stale candidates should no local client register a candidate in the new incarnation. The unit tests revealed an orthogonal issue with the PreLeader state. The PreLeader switches to Leader when the commit index is up to date but before applying the entries to the state. However the EOS may commit modifications immediately before the ApplyState message for prior entries is received. This can result in the "Store tree X and candidate base Y differ" exception. So I modified the PreLeader behavior to switch to Leader when the last applied index is up to date. This makes sense b/c the PreLeader bevavior is intended to protect the state from inconsistencies. I also fixed a couple bugs where the downPeerMemberNames was accessed with a String rather than a MemberName instance. This was a remnant of changing downPeerMemberNames to store MemberName. Change-Id: I326660c172353539146a2216cc8a70a4b842affe Signed-off-by: Tom Pantelis ---