Bug 4105: Change ownership on member down/up 04/26804/1
authorTom Pantelis <tpanteli@brocade.com>
Sat, 15 Aug 2015 21:23:50 +0000 (17:23 -0400)
committerTom Pantelis <tpanteli@brocade.com>
Thu, 10 Sep 2015 19:19:47 +0000 (15:19 -0400)
commit02f738dec4a31bdad04e42b2c19ecf09aacc0b87
tree964399c23098bf1ba4115c3d4fac2bf4ec8c33ef
parent24fa269a238e033e427f57a52fd15227975af9b6
Bug 4105: Change ownership on member down/up

Added 2 new messages, PeerUp and PeerDown, that the ShardManager sends
in response to cluster member events.

For PeerDown, the EntityOwnershipShard finds the entities owned by the
down member and selects a new owner based on the remaining candidates.
If there's no other candidates, the owner is cleared (set to "") so new
candidates can become owner. The down members are also tracked via a
downPeerMemberNames set.

For PeerUp, if the up member is in the downPeerMemberNames, the
EntityOwnershipShard finds entities that previously had their owner
cleared and attempts to select a new owner. This handles the case where
a previously down member was the only candidate for an entity so, when
that member comes back up, the entity's owner will be re-assigned to
that member.

Reassigning of owners via PeerDown and PeerUp is only done on the
leader. However that may not handle the case where the leader goes down.
When a new leader is elected we need it to select new owners for
entities owned by the down leader. There are 2 cases here. If the old
leader has not yet been detected as down then eventually we expect to
get PeerDown to handle it. The second case is if PeerDown was already
received prior to the leader change (probably the norm), in which case
PeerDown would not have been processed. To handle this case I overrode
onLeaderChanged to select new owners for entities owned by the old leader
that is passed in. The RaftActor sends the old leader's peerId so I
added a peerIdToMemberNames map to translate - this is populated via
PeerUp. Also I changed the RaftActor to track and pass the actual last valid
leader id, previously it passed the leader id from the previous behavior
which would normally be Candidate which always has a null leaderId.

The newOwner method was changed to ignore candidates in the
downPeerMemberNames set as there's no point in assigning the owner to a
candidate known to be down.

Change-Id: I8f0b78460a1a3e2a6418431f8a8a770a789e8f8d
Signed-off-by: Tom Pantelis <tpanteli@brocade.com>
13 files changed:
opendaylight/md-sal/sal-akka-raft/src/main/java/org/opendaylight/controller/cluster/raft/RaftActor.java
opendaylight/md-sal/sal-distributed-datastore/src/main/java/org/opendaylight/controller/cluster/datastore/ShardManager.java
opendaylight/md-sal/sal-distributed-datastore/src/main/java/org/opendaylight/controller/cluster/datastore/entityownership/DistributedEntityOwnershipService.java
opendaylight/md-sal/sal-distributed-datastore/src/main/java/org/opendaylight/controller/cluster/datastore/entityownership/EntityOwnersModel.java
opendaylight/md-sal/sal-distributed-datastore/src/main/java/org/opendaylight/controller/cluster/datastore/entityownership/EntityOwnershipShard.java
opendaylight/md-sal/sal-distributed-datastore/src/main/java/org/opendaylight/controller/cluster/datastore/entityownership/EntityOwnershipShardCommitCoordinator.java
opendaylight/md-sal/sal-distributed-datastore/src/main/java/org/opendaylight/controller/cluster/datastore/messages/PeerDown.java [new file with mode: 0644]
opendaylight/md-sal/sal-distributed-datastore/src/main/java/org/opendaylight/controller/cluster/datastore/messages/PeerUp.java [new file with mode: 0644]
opendaylight/md-sal/sal-distributed-datastore/src/main/yang/entity-owners.yang
opendaylight/md-sal/sal-distributed-datastore/src/test/java/org/opendaylight/controller/cluster/datastore/ShardManagerTest.java
opendaylight/md-sal/sal-distributed-datastore/src/test/java/org/opendaylight/controller/cluster/datastore/entityownership/EntityOwnershipShardTest.java
opendaylight/md-sal/sal-distributed-datastore/src/test/java/org/opendaylight/controller/cluster/datastore/utils/MockClusterWrapper.java
opendaylight/md-sal/sal-distributed-datastore/src/test/resources/simplelogger.properties