Refactor OVSDB Cluster test

- Add entity-owner shard check.
- Increase shard timers so we know when there is delay or real failure.
- Continue on entity-owner failure to update candidate list.
- Small fixes to suites variables.

Change-Id: If809be266a3fcc7d02bca284b9637f65ad8f1ce0
Signed-off-by: Luis Gomez <ecelgp@gmail.com>

diff --git a/csit/libraries/ClusterOvsdb.robot b/csit/libraries/ClusterOvsdb.robot
index 9737dc55de1be4ddbea273185ba5c04f13d201a5..945ac7f7ecf5c439ee15bc1a05d5dd3bf3050d41 100644 (file)
--- a/csit/libraries/ClusterOvsdb.robot
+++ b/csit/libraries/ClusterOvsdb.robot
@@ -11,12 +11,17 @@ Variables         ../variables/Variables.py
 Check Ovsdb Shards Status
     [Arguments]    ${controller_index_list}
     [Documentation]    Check Status for all shards in Ovsdb application.
-    ${topo_conf_leader}    ${topo_conf_followers_list}    Wait Until Keyword Succeeds    10s    1s    Get Cluster Shard Status    ${controller_index_list}
-    ...    config    topology
-    ${topo_oper_leader}    ${topo_oper_followers_list}    Wait Until Keyword Succeeds    10s    1s    Get Cluster Shard Status    ${controller_index_list}
-    ...    operational    topology
+    ${topo_conf_leader}    ${topo_conf_followers_list}    Get Cluster Shard Status    ${controller_index_list}    config    topology
+    ${topo_oper_leader}    ${topo_oper_followers_list}    Get Cluster Shard Status    ${controller_index_list}    operational    topology
+    ${owner_oper_leader}    ${owner_oper_followers_list}    Get Cluster Shard Status    ${controller_index_list}    operational    entity-ownership
     Log    config topology Leader is ${topo_conf_leader} and followers are ${topo_conf_followers_list}
     Log    operational topology Leader is ${topo_oper_leader} and followers are ${topo_oper_followers_list}
+    Log    operational entity-ownership Leader is ${owner_oper_leader} and followers are ${owner_oper_followers_list}
+
+Check Ovsdb Shards Status After Cluster Event
+    [Arguments]    ${controller_index_list}
+    [Documentation]    Check Shard Status after some cluster event.
+    Wait Until Keyword Succeeds    90s    1s    Check Ovsdb Shards Status    ${controller_index_list}
 
 Get Ovsdb Entity Owner Status For One Device
     [Arguments]    ${controller_index_list}
@@ -56,7 +61,6 @@ Get Cluster Entity Owner For Ovsdb
     : FOR    ${entity_candidate}    IN    @{entity_candidates_list}
     \    ${candidate}=    Replace String    &{entity_candidate}[name]    member-    ${EMPTY}
     \    ${candidate}=    Convert To Integer    ${candidate}
-    \    List Should Contain Value    ${controller_index_list}    ${candidate}    Candidate ${candidate} not exisiting in ${controller_index_list}
     \    Run Keyword If    '${candidate}' != '${owner}'    Append To List    ${candidates_list}    ${candidate}
     [Return]    ${owner}    ${candidates_list}
 
@@ -64,7 +68,7 @@ Create Bridge And Verify
     [Arguments]    ${controller_index_list}    ${controller_index}
     [Documentation]    Create bridge in ${controller_index} and verify it gets applied in all instances in ${controller_index_list}.
      # need to get UUID which should be the same on all controllers in cluster, so asking controller1
-    ${ovsdb_uuid}=    Get OVSDB UUID    controller_http_session=controller1
+    ${ovsdb_uuid}=    Get OVSDB UUID    controller_http_session=controller${controller_index}
     ${body}=    OperatingSystem.Get File    ${CURDIR}/../variables/ovsdb/create_bridge_3node.json
     ${body}    Replace String    ${body}    ovsdb://127.0.0.1:61644    ovsdb://uuid/${ovsdb_uuid}
     ${body}    Replace String    ${body}    tcp:controller1:6633    tcp:${ODL_SYSTEM_1_IP}:6640

diff --git a/csit/suites/ovsdb/Southbound_Cluster/010__Ovsdb_Southbound_Cluster.robot b/csit/suites/ovsdb/Southbound_Cluster/010__Ovsdb_Southbound_Cluster.robot
index 2a830f2f5838767d8c269e7f0a874d5b88a3a7db..27385d75bee944144fb7691a5a361687aeda2008 100644 (file)
--- a/csit/suites/ovsdb/Southbound_Cluster/010__Ovsdb_Southbound_Cluster.robot
+++ b/csit/suites/ovsdb/Southbound_Cluster/010__Ovsdb_Southbound_Cluster.robot
@@ -64,11 +64,13 @@ Kill Owner Instance
 
 Check Shards Status After Fail
     [Documentation]    Create original cluster list and check Status for all shards in Ovsdb application.
-    Wait Until Keyword Succeeds    10s    1s    Check Ovsdb Shards Status    ${new_cluster_list}
+    Check Ovsdb Shards Status After Cluster Event    ${new_cluster_list}
 
 Check Entity Owner Status And Find Owner and Candidate After Fail
     [Documentation]    Check Entity Owner Status and identify owner and candidate.
-    ${new_owner}    ${new_candidates_list}    Wait Until Keyword Succeeds    10s    1s    Get Ovsdb Entity Owner Status For One Device    ${new_cluster_list}
+    ${new_owner}    ${new_candidates_list}    Get Ovsdb Entity Owner Status For One Device    ${new_cluster_list}
+    Run Keyword And Continue On Failure    List Should Not Contain Value    ${new_candidates_list}    ${original_owner}    Original owner ${original_owner} still in candidate list.
+    Remove Values From List    ${new_candidates_list}    ${original_owner}
     ${new_candidate}=    Get From List    ${new_candidates_list}    0
     Set Suite Variable    ${new_owner}
     Set Suite Variable    ${new_candidate}
@@ -79,7 +81,7 @@ Create Bridge Manually In Owner and Verify After Fail
 
 Add Port Manually In Owner and Verify After Fail
     [Documentation]    Add Port in Owner and verify it gets applied from all instances.
-    Add Port To The Manual Bridge And Verify    ${original_cluster_list}    ${original_owner}
+    Add Port To The Manual Bridge And Verify    ${new_cluster_list}    ${new_owner}
 
 Create Bridge Via Controller In Owner and Verify After Fail
     [Documentation]    Create Bridge in Owner and verify it gets applied from all instances.
@@ -99,11 +101,11 @@ Start Old Owner Instance
 
 Check Shards Status After Recover
     [Documentation]    Create original cluster list and check Status for all shards in Ovsdb application.
-    Wait Until Keyword Succeeds    10s    1s    Check Ovsdb Shards Status    ${original_cluster_list}
+    Check Ovsdb Shards Status After Cluster Event    ${original_cluster_list}
 
 Check Entity Owner Status After Recover
     [Documentation]    Check Entity Owner Status and identify owner and candidate.
-    ${new_owner}    ${new_candidates_list}    Wait Until Keyword Succeeds    10s    1s    Get Ovsdb Entity Owner Status For One Device    ${original_cluster_list}
+    ${new_owner}    ${new_candidates_list}    Get Ovsdb Entity Owner Status For One Device    ${original_cluster_list}
     Set Suite Variable    ${new_owner}
 
 Create Bridge Manually In Owner and Verify After Recover
@@ -112,7 +114,7 @@ Create Bridge Manually In Owner and Verify After Recover
 
 Add Port Manually In Owner and Verify After Recover
     [Documentation]    Add Port in Owner and verify it gets applied from all instances.
-    Add Port To The Manual Bridge And Verify    ${original_cluster_list}    ${original_owner}
+    Add Port To The Manual Bridge And Verify    ${original_cluster_list}    ${new_owner}
 
 Create Bridge Via Controller In Owner and Verify After Recover
     [Documentation]    Create Bridge in Owner and verify it gets applied from all instances.