Bug 7391: Fix out-of-order LeaderStateChange events
On leader transition, the current leader first sends out
LeaderTransitioning events to each follower to tell them the
current leader is being deposed. The followers send out a
LeaderStateChange event with a null leaderId which is picked
up by the ShardManager to delay subsequent transactions activity
to the shard until a new leader is elected. However it's possible
the LeaderStateChange message does not reach a follower until after
the leader transition occurs (eg due to dispatching delay in the
caller or the network). This results in a LeaderStateChange event
with a null leaderId being delivered after the LeaderStateChange
with the new leaderId. I wrote a unit test that reproduces it.
We need to handle LeaderTransitioning events in a CAS-like manner,
ie include the leaderId with the LeaderTransitioning message and
only issue the LeaderStateChange event with a null leaderId if the
current leaderId matches the leaderId in the LeaderTransitioning
message.
Change-Id: I24e8bbf7707858ac4ed62f3a979cc0403daff8ac
Signed-off-by: Tom Pantelis <tpanteli@brocade.com>