*** Settings ***
Documentation Test suite for 2Node Cluster HA with Bulk Flows - Cluster node convergance and Data consistency after leader and follower restart with one switch connected
-Suite Setup Create Controller Sessions
+Suite Setup ClusterManagement Setup
Suite Teardown Delete All Sessions
Resource ../../../libraries/BulkomaticKeywords.robot
Resource ../../../libraries/MininetKeywords.robot
-Resource ../../../libraries/ClusterKeywords.robot
+Resource ../../../libraries/ClusterManagement.robot
Resource ../../../libraries/ClusterOpenFlow.robot
Resource ../../../libraries/Utils.robot
Variables ../../../variables/Variables.py
${operation_timeout} 100s
${restart_timeout} 350s
${flow_count_per_switch} 1000
-${switch_count_per_node} 1
+${switch_count} 1
${flow_count_after_add} 1000
${flow_count_after_del} 0
-${add_small_config} sal_add_bulk_flow_small_config.json
-${get_small_config} sal_get_bulk_flow_small_config.json
-${del_small_config} sal_del_bulk_flow_small_config.json
+${orig_json_config_add} sal_add_bulk_flow_config.json
+${orig_json_config_get} sal_get_bulk_flow_config.json
+${orig_json_config_del} sal_del_bulk_flow_config.json
*** Test Cases ***
-Create Original Cluster List
- [Documentation] Create original cluster list.
- ${original_cluster_list} ClusterKeywords.Create Controller Index List
- Set Suite Variable ${original_cluster_list}
-
-Check Shards Status Before Leader Restart
+Check Shards Status and Initialize Variables
[Documentation] Check Status for all shards in OpenFlow application.
- ClusterOpenFlow.Check OpenFlow Shards Status ${original_cluster_list}
+ ClusterOpenFlow.Check OpenFlow Shards Status
+ ${temp_json_config_add} BulkomaticKeywords.Set DPN And Flow Count In Json Add ${orig_json_config_add} ${switch_count} ${flow_count_per_switch}
+ ${temp_json_config_get} BulkomaticKeywords.Set DPN And Flow Count In Json Get ${orig_json_config_get} ${switch_count} ${flow_count_after_add}
+ ${temp_json_config_del} BulkomaticKeywords.Set DPN And Flow Count In Json Del ${orig_json_config_del} ${switch_count} ${flow_count_per_switch}
+ Set Suite Variable ${temp_json_config_add}
+ Set Suite Variable ${temp_json_config_get}
+ Set Suite Variable ${temp_json_config_del}
Get Inventory Follower Before Cluster Restart
[Documentation] Find a follower in the inventory config shard
- ${inventory_leader} ${inventory_followers} ClusterOpenFlow.Get InventoryConfig Shard Status ${original_cluster_list}
+ ${inventory_leader} ${inventory_followers} ClusterOpenFlow.Get InventoryConfig Shard Status
${Inventory_Leader_List}= Create List ${Inventory_Leader}
${Follower_Node_1}= Get From List ${Inventory_Followers} 0
Set Suite Variable ${Inventory_Followers}
Shutdown Leader From Cluster Node
[Documentation] Shutdown Leader Node and Start it Up.
- ClusterKeywords.Kill Multiple Controllers ${Inventory_Leader}
+ ClusterManagement.Kill Single Member ${Inventory_Leader}
Check Shards Status After Leader Shutdown
[Documentation] Wait for node convergence and check status for all shards in OpenFlow application.
Add Bulk Flow From Follower
[Documentation] 1000 Flows added via Follower Node1 and verify it gets applied in all instances.
- BulkomaticKeywords.Add Bulk Flow In Node ${Follower_Node_1} ${add_small_config} ${operation_timeout}
+ BulkomaticKeywords.Add Bulk Flow In Node ${temp_json_config_add} ${Follower_Node_1} ${operation_timeout}
Get Bulk Flows and Verify In Cluster
- BulkomaticKeywords.Get Bulk Flow And Verify Count In Cluster ${Inventory_Followers} ${get_small_config} ${operation_timeout} ${flow_count_after_add}
+ [Documentation] Initiate get operation and check flow count across cluster nodes.
+ BulkomaticKeywords.Get Bulk Flow And Verify Count In Cluster ${temp_json_config_get} ${operation_timeout} ${flow_count_after_add} ${Inventory_Followers}
Verify Flows In Switch Before Cluster Restart
[Documentation] Verify flows are installed in switch before cluster restart.
- MininetKeywords.Verify Aggregate Flow From Mininet Session ${mininet_conn_id} ${switch_count_per_node} ${flow_count_per_switch} ${operation_timeout}
+ MininetKeywords.Verify Aggregate Flow From Mininet Session ${mininet_conn_id} ${flow_count_after_add} ${operation_timeout}
Restart Leader From Cluster Node
[Documentation] Restart Leader Node.
- ClusterKeywords.Start Multiple Controllers ${operation_timeout} ${Inventory_Leader}
+ ClusterManagement.Start Single Member ${Inventory_Leader}
Verify Data Recovery After Leader Restart
[Documentation] 1000 Flows preserved in all controller instances.
- Wait Until Keyword Succeeds ${restart_timeout} 2s BulkomaticKeywords.Get Bulk Flow And Verify Count In Cluster ${original_cluster_list} ${get_small_config} ${operation_timeout}
- ... ${flow_count_after_add}
+ Wait Until Keyword Succeeds ${restart_timeout} 2s BulkomaticKeywords.Get Bulk Flow And Verify Count In Cluster ${temp_json_config_get} ${operation_timeout} ${flow_count_after_add}
Verify Flows In Switch After Leader Restart
[Documentation] Verify flows are installed in switch after cluster restart.
- MininetKeywords.Verify Aggregate Flow From Mininet Session ${mininet_conn_id} ${switch_count_per_node} ${flow_count_per_switch} ${operation_timeout}
+ MininetKeywords.Verify Aggregate Flow From Mininet Session ${mininet_conn_id} ${flow_count_after_add} ${operation_timeout}
Stop Mininet Connected To Follower Node1
[Documentation] Stop mininet and exit connection.
Utils.Clean Mininet System
Delete All Flows From Follower Node1
- [Documentation] Rest 1000 Flows deleted via Follower Node1 and verify it gets applied in all instances.
- BulkomaticKeywords.Delete Bulk Flow In Node ${Follower_Node_1} ${del_small_config} ${operation_timeout}
+ [Documentation] 1000 Flows deleted via Follower Node1 and verify it gets applied in all instances.
+ BulkomaticKeywords.Delete Bulk Flow In Node ${temp_json_config_del} ${Follower_Node_1} ${operation_timeout}
Verify No Flows In Cluster
- BulkomaticKeywords.Get Bulk Flow And Verify Count In Cluster ${original_cluster_list} ${get_small_config} ${operation_timeout} ${flow_count_after_del}
+ [Documentation] Verify flow count is 0 across cluster nodes.
+ BulkomaticKeywords.Get Bulk Flow And Verify Count In Cluster ${temp_json_config_get} ${operation_timeout} ${flow_count_after_del}
Get Inventory Follower Before Follower Restart
[Documentation] Find Leader and followers in the inventory config shard
- ${inventory_leader} ${inventory_followers} ClusterOpenFlow.Get InventoryConfig Shard Status ${original_cluster_list}
+ ${inventory_leader} ${inventory_followers} ClusterOpenFlow.Get InventoryConfig Shard Status
${Active_Nodes}= Create List
${Follower_Node_1}= Get From List ${Inventory_Followers} 0
${Follower_Node_2}= Get From List ${Inventory_Followers} 1
Shutdown Follower From Cluster Node
[Documentation] Shutdown Follower Node2 and Start it Up.
- ClusterKeywords.Kill Multiple Controllers ${Follower_Node_2}
+ ClusterManagement.Kill Single Member ${Follower_Node_2}
Check Shards Status After Follower Shutdown
[Documentation] Wait for node convergence and check status for all shards in OpenFlow application.
Add Bulk Flow From Follower Node1
[Documentation] 1000 Flows added via Follower Node1 and verify it gets applied in all instances.
- BulkomaticKeywords.Add Bulk Flow In Node ${Follower_Node_1} ${add_small_config} ${operation_timeout}
+ BulkomaticKeywords.Add Bulk Flow In Node ${temp_json_config_add} ${Follower_Node_1} ${operation_timeout}
Get Bulk Flows and Verify In Cluster Before Follower Restart
- BulkomaticKeywords.Get Bulk Flow And Verify Count In Cluster ${Active_Nodes} ${get_small_config} ${operation_timeout} ${flow_count_after_add}
+ [Documentation] Initiate get operation and check flow count only across active cluster nodes
+ BulkomaticKeywords.Get Bulk Flow And Verify Count In Cluster ${temp_json_config_get} ${operation_timeout} ${flow_count_after_add} ${Active_Nodes}
Verify Flows In Switch Before Follower Restart
[Documentation] Verify flows are installed in switch before follower restart.
- MininetKeywords.Verify Aggregate Flow From Mininet Session ${mininet_conn_id} ${switch_count_per_node} ${flow_count_per_switch} ${operation_timeout}
+ MininetKeywords.Verify Aggregate Flow From Mininet Session ${mininet_conn_id} ${flow_count_after_add} ${operation_timeout}
Restart Follower From Cluster Node
[Documentation] Restart Follower Node2.
- ClusterKeywords.Start Multiple Controllers ${operation_timeout} ${Follower_Node_2}
+ ClusterManagement.Start Single Member ${Follower_Node_2}
Verify Data Recovery After Follower Restart
[Documentation] 1000 Flows preserved in all controller instances.
- Wait Until Keyword Succeeds ${restart_timeout} 2s BulkomaticKeywords.Get Bulk Flow And Verify Count In Cluster ${original_cluster_list} ${get_small_config} ${operation_timeout}
- ... ${flow_count_after_add}
+ Wait Until Keyword Succeeds ${restart_timeout} 2s BulkomaticKeywords.Get Bulk Flow And Verify Count In Cluster ${temp_json_config_get} ${operation_timeout} ${flow_count_after_add}
Verify Flows In Switch After Follower Restart
[Documentation] Verify flows are installed in switch after cluster restart.
- MininetKeywords.Verify Aggregate Flow From Mininet Session ${mininet_conn_id} ${switch_count_per_node} ${flow_count_per_switch} ${operation_timeout}
+ MininetKeywords.Verify Aggregate Flow From Mininet Session ${mininet_conn_id} ${flow_count_after_add} ${operation_timeout}
Stop Mininet Connected To Follower Node
[Documentation] Stop mininet and exit connection.
Utils.Clean Mininet System
Delete All Flows From Follower Node
- [Documentation] Rest 1000 Flows deleted via Follower Node1 and verify it gets applied in all instances.
- BulkomaticKeywords.Delete Bulk Flow In Node ${Follower_Node_1} ${del_small_config} ${operation_timeout}
+ [Documentation] 1000 Flows deleted via Follower Node1 and verify it gets applied in all instances.
+ BulkomaticKeywords.Delete Bulk Flow In Node ${temp_json_config_del} ${Follower_Node_1} ${operation_timeout}
Verify No Flows In Cluster After Follower Restart
- BulkomaticKeywords.Get Bulk Flow And Verify Count In Cluster ${original_cluster_list} ${get_small_config} ${operation_timeout} ${flow_count_after_del}
+ [Documentation] Verify flow count is 0 across cluster nodes.
+ BulkomaticKeywords.Get Bulk Flow And Verify Count In Cluster ${temp_json_config_get} ${operation_timeout} ${flow_count_after_del}