Add fail if leader's election take much time

After the leader isolation, remaining nodes elect new leader.
If new leader election verification takes more time than
transaction timout, the suite will not be able to test
a case with cluster healing within transaction timeout. In this case
a fail is added.

Change-Id: I36cfabf2969b6f51f82eeb8032d6365bcb61a165
Signed-off-by: Peter Gubka <pgubka@cisco.com>

diff --git a/csit/libraries/controller/DdbCommons.robot b/csit/libraries/controller/DdbCommons.robot
index c203504a9eb6cfc99be2205f2c0ad1815b7d0ce1..b6c29e7c5a53af193b80a55809c32d110e0e9e09 100644 (file)
--- a/csit/libraries/controller/DdbCommons.robot
+++ b/csit/libraries/controller/DdbCommons.robot
@@ -36,6 +36,7 @@ ${HARD_TIMEOUT}    ${2*${TRANSACTION_TIMEOUT}}
 ${PREF_BASED_SHARD}    id-ints
 ${TEST_LOG_LEVEL}    info
 @{TEST_LOG_COMPONENTS}    org.opendaylight.controller.cluster.sharding    org.opendaylight.controller.cluster.datastore
+${HEAL_WITHIN_TRANS_TIMEOUT}    ${0}
 
 *** Keywords ***
 Explicit_Leader_Movement_Test_Templ
@@ -140,6 +141,12 @@ Leader_Isolation_Test_Templ
     ${li_isolated}    BuiltIn.Set_Variable    ${True}
     BuiltIn.Wait_Until_Keyword_Succeeds    45s    2s    ClusterManagement.Verify_Shard_Leader_Elected    ${shard_name}    ${shard_type}    ${True}
     ...    ${leader}    member_index_list=${follower_list}
+    ${date_leader_elected} =    DateTime.Get_Current_Date
+    ${delta} =    DateTime.Subtract_Date_From_Date    ${date_leader_elected}    ${date_start}
+    # TODO: Consider extracting the block which depends on ${heal_timeout} value into two keywords, and have just one If to decide which one to call.
+    BuiltIn.Run_Keyword_If    ${heal_timeout}==${HEAL_WITHIN_TRANS_TIMEOUT} and ${delta} > ${TRANSACTION_TIMEOUT}    BuiltIn.Fail    We are not able to heal the cluster within transaction timout because new leader election took more time.
+    ${resp} =    MdsalLowlevelPy.Get_Next_Transactions_Response
+    BuiltIn.Should_Be_Equal    ${resp}    ${NONE}    No response expected, received ${resp}
     BuiltIn.Sleep    ${heal_timeout}
     ClusterManagement.Rejoin_Member_From_List_Or_All    ${leader}
     ${li_isolated}    BuiltIn.Set_Variable    ${False}
@@ -166,6 +173,12 @@ Leader_Isolation_PrefBasedShard_Test_Templ
     ${li_isolated}    BuiltIn.Set_Variable    ${True}
     BuiltIn.Wait_Until_Keyword_Succeeds    45s    2s    ClusterManagement.Verify_Shard_Leader_Elected    ${shard_name}!!    ${shard_type}    ${True}
     ...    ${leader}    member_index_list=${follower_list}
+    ${date_leader_elected} =    DateTime.Get_Current_Date
+    ${delta} =    DateTime.Subtract_Date_From_Date    ${date_leader_elected}    ${date_start}
+    # TODO: Consider extracting the block which depends on ${heal_timeout} value into two keywords, and have just one If to decide which one to call.
+    BuiltIn.Run_Keyword_If    ${heal_timeout}==${HEAL_WITHIN_TRANS_TIMEOUT} and ${delta} > ${TRANSACTION_TIMEOUT}    BuiltIn.Fail    We are not able to heal the cluster within transaction timout because new leader election took more time.
+    ${resp} =    MdsalLowlevelPy.Get_Next_Transactions_Response
+    BuiltIn.Should_Be_Equal    ${resp}    ${NONE}    No response expected, received ${resp}
     BuiltIn.Sleep    ${heal_timeout}
     ClusterManagement.Rejoin_Member_From_List_Or_All    ${leader}
     ${li_isolated}    BuiltIn.Set_Variable    ${False}

diff --git a/csit/suites/controller/dom_data_broker/leader_isolation.robot b/csit/suites/controller/dom_data_broker/leader_isolation.robot
index d49362a5473fcdae5479c34e476aa8ecf2d91070..510362cd457649984df9c90fe06fe262251e3872 100644 (file)
--- a/csit/suites/controller/dom_data_broker/leader_isolation.robot
+++ b/csit/suites/controller/dom_data_broker/leader_isolation.robot
@@ -23,7 +23,7 @@ Resource          ${CURDIR}/../../../libraries/SetupUtils.robot
 Healing_Within_Transaction_Timeout
     [Documentation]    The isolated node (leader) is rejoined as soon as new leader is elected and
     ...    and within transaction timeout.
-    ${0}
+    ${HEAL_WITHIN_TRANS_TIMEOUT}
 
 Healing_After_2x_Transaction_Timeout
     [Documentation]    The isolated node (leader) is rejoined after 2x transaction timeout.

diff --git a/csit/suites/controller/dom_data_broker/leader_isolation_prefbasedshard.robot b/csit/suites/controller/dom_data_broker/leader_isolation_prefbasedshard.robot
index 3acb2dd7d01e04540e78dec3f9366b31a18d3eb3..0108e841cf23fd5f596a346557447dde2400a43c 100644 (file)
--- a/csit/suites/controller/dom_data_broker/leader_isolation_prefbasedshard.robot
+++ b/csit/suites/controller/dom_data_broker/leader_isolation_prefbasedshard.robot
@@ -25,7 +25,7 @@ Resource          ${CURDIR}/../../../libraries/SetupUtils.robot
 Healing_Within_Transaction_Timeout
     [Documentation]    The isolated node (leader) is rejoined as soon as new leader is elected and
     ...    and within transaction timeout.
-    ${0}
+    ${HEAL_WITHIN_TRANS_TIMEOUT}
 
 Healing_After_2x_Transaction_Timeout
     [Documentation]    The isolated node (leader) is rejoined after 2x transaction timeout.