Node Convergance and Data Consistency in 2 Node Cluster

Description: In a 3 node cluster intial inventory shard status is
verified and following tests are performed.

- Shutdown Leader node and in 2Node system connect mininet to follower
  node and push bulk flow in that node

- Verify data consistency across 2 Active Nodes

- Restart Leader and verify data consistency after node convergance

- Verify flow count in switch

- Similarly test for follower shutdown case

Change-Id: Id5714ee52e5956eaf546e34dae738a8d996f255a
Signed-off-by: Sanjib Mohapatra <sanjib.mohapatra@ericsson.com>

diff --git a/csit/suites/openflowplugin/Clustering_Bulkomatic/020__Cluster_HA_Data_Recovery_BulkFlow_2Node_Cluster.robot b/csit/suites/openflowplugin/Clustering_Bulkomatic/020__Cluster_HA_Data_Recovery_BulkFlow_2Node_Cluster.robot
new file mode 100644 (file)
index 0000000..f26a830
--- /dev/null
+++ b/csit/suites/openflowplugin/Clustering_Bulkomatic/020__Cluster_HA_Data_Recovery_BulkFlow_2Node_Cluster.robot
@@ -0,0 +1,150 @@
+*** Settings ***
+Documentation     Test suite for 2Node Cluster HA with Bulk Flows - Cluster node convergance and Data consistency after leader and follower restart with one switch connected
+Suite Setup       Create Controller Sessions
+Suite Teardown    Delete All Sessions
+Resource          ../../../libraries/BulkomaticKeywords.robot
+Resource          ../../../libraries/MininetKeywords.robot
+Resource          ../../../libraries/ClusterKeywords.robot
+Resource          ../../../libraries/ClusterOpenFlow.robot
+Resource          ../../../libraries/Utils.robot
+Variables         ../../../variables/Variables.py
+
+*** Variables ***
+${operation_timeout}    100s
+${restart_timeout}    350s
+${flow_count_per_switch}    1000
+${switch_count_per_node}    1
+${flow_count_after_add}    1000
+${flow_count_after_del}    0
+${add_small_config}    sal_add_bulk_flow_small_config.json
+${get_small_config}    sal_get_bulk_flow_small_config.json
+${del_small_config}    sal_del_bulk_flow_small_config.json
+
+*** Test Cases ***
+Create Original Cluster List
+    [Documentation]    Create original cluster list.
+    ${original_cluster_list}    ClusterKeywords.Create Controller Index List
+    Set Suite Variable    ${original_cluster_list}
+
+Check Shards Status Before Leader Restart
+    [Documentation]    Check Status for all shards in OpenFlow application.
+    ClusterOpenFlow.Check OpenFlow Shards Status    ${original_cluster_list}
+
+Get Inventory Follower Before Cluster Restart
+    [Documentation]    Find a follower in the inventory config shard
+    ${inventory_leader}    ${inventory_followers}    ClusterOpenFlow.Get InventoryConfig Shard Status    ${original_cluster_list}
+    ${Inventory_Leader_List}=    Create List    ${Inventory_Leader}
+    ${Follower_Node_1}=    Get From List    ${Inventory_Followers}    0
+    Set Suite Variable    ${Inventory_Followers}
+    Set Suite Variable    ${Follower_Node_1}
+    Set Suite Variable    ${Inventory_Leader}
+    Set Suite Variable    ${Inventory_Leader_List}
+
+Shutdown Leader From Cluster Node
+    [Documentation]    Shutdown Leader Node and Start it Up.
+    ClusterKeywords.Kill Multiple Controllers    ${Inventory_Leader}
+
+Check Shards Status After Leader Shutdown
+    [Documentation]    Wait for node convergence and check status for all shards in OpenFlow application.
+    Wait Until Keyword Succeeds    ${operation_timeout}    2s    ClusterOpenFlow.Check OpenFlow Shards Status    ${Inventory_Followers}
+
+Start Mininet Connect To Follower Node1
+    [Documentation]    Start mininet with connection to Follower Node1.
+    ${mininet_conn_id}=    MininetKeywords.Start Mininet Single Controller    ${TOOLS_SYSTEM_IP}    ${ODL_SYSTEM_${Follower_Node_1}_IP}
+    Set Suite Variable    ${mininet_conn_id}
+
+Add Bulk Flow From Follower
+    [Documentation]    1000 Flows added via Follower Node1 and verify it gets applied in all instances.
+    BulkomaticKeywords.Add Bulk Flow In Node    ${Follower_Node_1}    ${add_small_config}    ${operation_timeout}
+
+Get Bulk Flows and Verify In Cluster
+    BulkomaticKeywords.Get Bulk Flow And Verify Count In Cluster    ${Inventory_Followers}    ${get_small_config}    ${operation_timeout}    ${flow_count_after_add}
+
+Verify Flows In Switch Before Cluster Restart
+    [Documentation]    Verify flows are installed in switch before cluster restart.
+    MininetKeywords.Verify Aggregate Flow From Mininet Session    ${mininet_conn_id}    ${switch_count_per_node}    ${flow_count_per_switch}    ${operation_timeout}
+
+Restart Leader From Cluster Node
+    [Documentation]    Restart Leader Node.
+    ClusterKeywords.Start Multiple Controllers    ${operation_timeout}    ${Inventory_Leader}
+
+Verify Data Recovery After Leader Restart
+    [Documentation]    1000 Flows preserved in all controller instances.
+    Wait Until Keyword Succeeds    ${restart_timeout}    2s    BulkomaticKeywords.Get Bulk Flow And Verify Count In Cluster    ${original_cluster_list}    ${get_small_config}    ${operation_timeout}
+    ...    ${flow_count_after_add}
+
+Verify Flows In Switch After Leader Restart
+    [Documentation]    Verify flows are installed in switch after cluster restart.
+    MininetKeywords.Verify Aggregate Flow From Mininet Session    ${mininet_conn_id}    ${switch_count_per_node}    ${flow_count_per_switch}    ${operation_timeout}
+
+Stop Mininet Connected To Follower Node1
+    [Documentation]    Stop mininet and exit connection.
+    MininetKeywords.Stop Mininet And Exit    ${mininet_conn_id}
+    Utils.Clean Mininet System
+
+Delete All Flows From Follower Node1
+    [Documentation]    Rest 1000 Flows deleted via Follower Node1 and verify it gets applied in all instances.
+    BulkomaticKeywords.Delete Bulk Flow In Node    ${Follower_Node_1}    ${del_small_config}    ${operation_timeout}
+
+Verify No Flows In Cluster
+    BulkomaticKeywords.Get Bulk Flow And Verify Count In Cluster    ${original_cluster_list}    ${get_small_config}    ${operation_timeout}    ${flow_count_after_del}
+
+Get Inventory Follower Before Follower Restart
+    [Documentation]    Find Leader and followers in the inventory config shard
+    ${inventory_leader}    ${inventory_followers}    ClusterOpenFlow.Get InventoryConfig Shard Status    ${original_cluster_list}
+    ${Active_Nodes}=    Create List
+    ${Follower_Node_1}=    Get From List    ${Inventory_Followers}    0
+    ${Follower_Node_2}=    Get From List    ${Inventory_Followers}    1
+    Append To List    ${Active_Nodes}    ${inventory_leader}    ${Follower_Node_1}
+    Set Suite Variable    ${Active_Nodes}
+    Set Suite Variable    ${Follower_Node_1}
+    Set Suite Variable    ${Follower_Node_2}
+
+Shutdown Follower From Cluster Node
+    [Documentation]    Shutdown Follower Node2 and Start it Up.
+    ClusterKeywords.Kill Multiple Controllers    ${Follower_Node_2}
+
+Check Shards Status After Follower Shutdown
+    [Documentation]    Wait for node convergence and check status for all shards in OpenFlow application.
+    Wait Until Keyword Succeeds    ${operation_timeout}    2s    ClusterOpenFlow.Check OpenFlow Shards Status    ${Active_Nodes}
+
+Start Mininet Connect To Follower Node
+    [Documentation]    Start mininet with connection to Follower Node1.
+    ${mininet_conn_id}=    MininetKeywords.Start Mininet Single Controller    ${TOOLS_SYSTEM_IP}    ${ODL_SYSTEM_${Follower_Node_1}_IP}
+    Set Suite Variable    ${mininet_conn_id}
+
+Add Bulk Flow From Follower Node1
+    [Documentation]    1000 Flows added via Follower Node1 and verify it gets applied in all instances.
+    BulkomaticKeywords.Add Bulk Flow In Node    ${Follower_Node_1}    ${add_small_config}    ${operation_timeout}
+
+Get Bulk Flows and Verify In Cluster Before Follower Restart
+    BulkomaticKeywords.Get Bulk Flow And Verify Count In Cluster    ${Active_Nodes}    ${get_small_config}    ${operation_timeout}    ${flow_count_after_add}
+
+Verify Flows In Switch Before Follower Restart
+    [Documentation]    Verify flows are installed in switch before follower restart.
+    MininetKeywords.Verify Aggregate Flow From Mininet Session    ${mininet_conn_id}    ${switch_count_per_node}    ${flow_count_per_switch}    ${operation_timeout}
+
+Restart Follower From Cluster Node
+    [Documentation]    Restart Follower Node2.
+    ClusterKeywords.Start Multiple Controllers    ${operation_timeout}    ${Follower_Node_2}
+
+Verify Data Recovery After Follower Restart
+    [Documentation]    1000 Flows preserved in all controller instances.
+    Wait Until Keyword Succeeds    ${restart_timeout}    2s    BulkomaticKeywords.Get Bulk Flow And Verify Count In Cluster    ${original_cluster_list}    ${get_small_config}    ${operation_timeout}
+    ...    ${flow_count_after_add}
+
+Verify Flows In Switch After Follower Restart
+    [Documentation]    Verify flows are installed in switch after cluster restart.
+    MininetKeywords.Verify Aggregate Flow From Mininet Session    ${mininet_conn_id}    ${switch_count_per_node}    ${flow_count_per_switch}    ${operation_timeout}
+
+Stop Mininet Connected To Follower Node
+    [Documentation]    Stop mininet and exit connection.
+    MininetKeywords.Stop Mininet And Exit    ${mininet_conn_id}
+    Utils.Clean Mininet System
+
+Delete All Flows From Follower Node
+    [Documentation]    Rest 1000 Flows deleted via Follower Node1 and verify it gets applied in all instances.
+    BulkomaticKeywords.Delete Bulk Flow In Node    ${Follower_Node_1}    ${del_small_config}    ${operation_timeout}
+
+Verify No Flows In Cluster After Follower Restart
+    BulkomaticKeywords.Get Bulk Flow And Verify Count In Cluster    ${original_cluster_list}    ${get_small_config}    ${operation_timeout}    ${flow_count_after_del}