Fix intermittent failure in testLeadershipTransferOnShutdown

10:03:06 java.util.concurrent.ExecutionException: ReadFailedException{message=Error executeRead ReadData for path /(urn:opendaylight:params:xml:ns:yang:controller:md:sal:dom:store:test:cars?revision=2014-03-13)cars/car, errorList=[RpcError [message=Error executeRead ReadData for path /(urn:opendaylight:params:xml:ns:yang:controller:md:sal:dom:store:test:cars?revision=2014-03-13)cars/car, severity=ERROR, errorType=APPLICATION, tag=operation-failed, applicationTag=null, info=null, cause=org.opendaylight.controller.md.sal.common.api.data.DataStoreUnavailableException: Shard member-2-shard-cars-testLeadershipTransferOnShutdown currently has no leader. Try again later.]]}
10:03:06 	at org.opendaylight.yangtools.util.concurrent.MappingCheckedFuture.wrapInExecutionException(MappingCheckedFuture.java:64)
10:03:06 	at org.opendaylight.yangtools.util.concurrent.MappingCheckedFuture.get(MappingCheckedFuture.java:92)
10:03:06 	at org.opendaylight.controller.cluster.datastore.DistributedDataStoreRemotingIntegrationTest.verifyCars(DistributedDataStoreRemotingIntegrationTest.java:215)
10:03:06 	at org.opendaylight.controller.cluster.datastore.DistributedDataStoreRemotingIntegrationTest.testLeadershipTransferOnShutdown(DistributedDataStoreRemotingIntegrationTest.java:928)

From the logs it seems member-2 hadn't gotten MemberUp for member-3 after the
leader transfer and by the time it tried to read. I added calls to wait for members
to be up. After the change it ran 333 times w/o failure.

Change-Id: Ifbbf304230292f69429d3086867679effb8db01c
Signed-off-by: Tom Pantelis <tompantelis@gmail.com>

diff --git a/opendaylight/md-sal/sal-distributed-datastore/src/test/java/org/opendaylight/controller/cluster/datastore/DistributedDataStoreRemotingIntegrationTest.java b/opendaylight/md-sal/sal-distributed-datastore/src/test/java/org/opendaylight/controller/cluster/datastore/DistributedDataStoreRemotingIntegrationTest.java
index 1d4d66e57000e72c19aeae75ce583ba8ec245e41..63d785c984659a39c41664637e0fef5860259143 100644 (file)
--- a/opendaylight/md-sal/sal-distributed-datastore/src/test/java/org/opendaylight/controller/cluster/datastore/DistributedDataStoreRemotingIntegrationTest.java
+++ b/opendaylight/md-sal/sal-distributed-datastore/src/test/java/org/opendaylight/controller/cluster/datastore/DistributedDataStoreRemotingIntegrationTest.java
@@ -208,6 +208,9 @@ public class DistributedDataStoreRemotingIntegrationTest extends AbstractTest {
                 testParameter, type, moduleShardsConfig, false, shards);
 
         leaderTestKit.waitUntilLeader(leaderDistributedDataStore.getActorContext(), shards);
+
+        leaderTestKit.waitForMembersUp("member-2");
+        followerTestKit.waitForMembersUp("member-1");
     }
 
     private static void verifyCars(final DOMStoreReadTransaction readTx, final MapEntryNode... entries)
@@ -884,6 +887,9 @@ public class DistributedDataStoreRemotingIntegrationTest extends AbstractTest {
         try (AbstractDataStore follower2DistributedDataStore = follower2TestKit.setupAbstractDataStore(
                 testParameter, testName, MODULE_SHARDS_CARS_PEOPLE_1_2_3, false)) {
 
+            followerTestKit.waitForMembersUp("member-3");
+            follower2TestKit.waitForMembersUp("member-1", "member-2");
+
             // Create and submit a couple tx's so they're pending.
 
             DOMStoreWriteTransaction writeTx = followerDistributedDataStore.newWriteOnlyTransaction();