2 Documentation Test suite for 2Node Cluster HA with Bulk Flows - Cluster node convergance and Data consistency after leader and follower restart with one switch connected
3 Suite Setup Create Controller Sessions
4 Suite Teardown Delete All Sessions
5 Resource ../../../libraries/BulkomaticKeywords.robot
6 Resource ../../../libraries/MininetKeywords.robot
7 Resource ../../../libraries/ClusterKeywords.robot
8 Resource ../../../libraries/ClusterOpenFlow.robot
9 Resource ../../../libraries/Utils.robot
10 Variables ../../../variables/Variables.py
13 ${operation_timeout} 100s
14 ${restart_timeout} 350s
15 ${flow_count_per_switch} 1000
17 ${flow_count_after_add} 1000
18 ${flow_count_after_del} 0
19 ${orig_json_config_add} sal_add_bulk_flow_config.json
20 ${orig_json_config_get} sal_get_bulk_flow_config.json
21 ${orig_json_config_del} sal_del_bulk_flow_config.json
24 Create Original Cluster List
25 [Documentation] Create original cluster list.
26 ${original_cluster_list} ClusterKeywords.Create Controller Index List
27 Set Suite Variable ${original_cluster_list}
29 Check Shards Status Before Leader Restart
30 [Documentation] Check Status for all shards in OpenFlow application.
31 ClusterOpenFlow.Check OpenFlow Shards Status ${original_cluster_list}
33 Get Inventory Follower Before Cluster Restart
34 [Documentation] Find a follower in the inventory config shard
35 ${inventory_leader} ${inventory_followers} ClusterOpenFlow.Get InventoryConfig Shard Status ${original_cluster_list}
36 ${Inventory_Leader_List}= Create List ${Inventory_Leader}
37 ${Follower_Node_1}= Get From List ${Inventory_Followers} 0
38 Set Suite Variable ${Inventory_Followers}
39 Set Suite Variable ${Follower_Node_1}
40 Set Suite Variable ${Inventory_Leader}
41 Set Suite Variable ${Inventory_Leader_List}
43 Shutdown Leader From Cluster Node
44 [Documentation] Shutdown Leader Node and Start it Up.
45 ClusterKeywords.Kill Multiple Controllers ${Inventory_Leader}
47 Check Shards Status After Leader Shutdown
48 [Documentation] Wait for node convergence and check status for all shards in OpenFlow application.
49 Wait Until Keyword Succeeds ${operation_timeout} 2s ClusterOpenFlow.Check OpenFlow Shards Status ${Inventory_Followers}
51 Start Mininet Connect To Follower Node1
52 [Documentation] Start mininet with connection to Follower Node1.
53 ${mininet_conn_id}= MininetKeywords.Start Mininet Single Controller ${TOOLS_SYSTEM_IP} ${ODL_SYSTEM_${Follower_Node_1}_IP}
54 Set Suite Variable ${mininet_conn_id}
56 Add Bulk Flow From Follower
57 [Documentation] 1000 Flows added via Follower Node1 and verify it gets applied in all instances.
58 ${temp_json_config_add} BulkomaticKeywords.Set DPN And Flow Count In Json Add ${orig_json_config_add} ${switch_count} ${flow_count_per_switch}
59 Set Suite Variable ${temp_json_config_add}
60 BulkomaticKeywords.Add Bulk Flow In Node ${Follower_Node_1} ${temp_json_config_add} ${operation_timeout}
62 Get Bulk Flows and Verify In Cluster
63 [Documentation] Initiate get operation and check flow count across cluster nodes.
64 ${temp_json_config_get} BulkomaticKeywords.Set DPN And Flow Count In Json Get ${orig_json_config_get} ${switch_count} ${flow_count_after_add}
65 Set Suite Variable ${temp_json_config_get}
66 BulkomaticKeywords.Get Bulk Flow And Verify Count In Cluster ${Inventory_Followers} ${temp_json_config_get} ${operation_timeout} ${flow_count_after_add}
68 Verify Flows In Switch Before Cluster Restart
69 [Documentation] Verify flows are installed in switch before cluster restart.
70 MininetKeywords.Verify Aggregate Flow From Mininet Session ${mininet_conn_id} ${switch_count} ${flow_count_per_switch} ${operation_timeout}
72 Restart Leader From Cluster Node
73 [Documentation] Restart Leader Node.
74 ClusterKeywords.Start Multiple Controllers ${operation_timeout} ${Inventory_Leader}
76 Verify Data Recovery After Leader Restart
77 [Documentation] 1000 Flows preserved in all controller instances.
78 Wait Until Keyword Succeeds ${restart_timeout} 2s BulkomaticKeywords.Get Bulk Flow And Verify Count In Cluster ${original_cluster_list} ${temp_json_config_get} ${operation_timeout}
79 ... ${flow_count_after_add}
81 Verify Flows In Switch After Leader Restart
82 [Documentation] Verify flows are installed in switch after cluster restart.
83 MininetKeywords.Verify Aggregate Flow From Mininet Session ${mininet_conn_id} ${switch_count} ${flow_count_per_switch} ${operation_timeout}
85 Stop Mininet Connected To Follower Node1
86 [Documentation] Stop mininet and exit connection.
87 MininetKeywords.Stop Mininet And Exit ${mininet_conn_id}
88 Utils.Clean Mininet System
90 Delete All Flows From Follower Node1
91 [Documentation] 1000 Flows deleted via Follower Node1 and verify it gets applied in all instances.
92 ${temp_json_config_del} BulkomaticKeywords.Set DPN And Flow Count In Json Del ${orig_json_config_del} ${switch_count} ${flow_count_per_switch}
93 Set Suite Variable ${temp_json_config_del}
94 BulkomaticKeywords.Delete Bulk Flow In Node ${Follower_Node_1} ${temp_json_config_del} ${operation_timeout}
96 Verify No Flows In Cluster
97 [Documentation] Verify flow count is 0 across cluster nodes.
98 BulkomaticKeywords.Get Bulk Flow And Verify Count In Cluster ${original_cluster_list} ${temp_json_config_get} ${operation_timeout} ${flow_count_after_del}
100 Get Inventory Follower Before Follower Restart
101 [Documentation] Find Leader and followers in the inventory config shard
102 ${inventory_leader} ${inventory_followers} ClusterOpenFlow.Get InventoryConfig Shard Status ${original_cluster_list}
103 ${Active_Nodes}= Create List
104 ${Follower_Node_1}= Get From List ${Inventory_Followers} 0
105 ${Follower_Node_2}= Get From List ${Inventory_Followers} 1
106 Append To List ${Active_Nodes} ${inventory_leader} ${Follower_Node_1}
107 Set Suite Variable ${Active_Nodes}
108 Set Suite Variable ${Follower_Node_1}
109 Set Suite Variable ${Follower_Node_2}
111 Shutdown Follower From Cluster Node
112 [Documentation] Shutdown Follower Node2 and Start it Up.
113 ClusterKeywords.Kill Multiple Controllers ${Follower_Node_2}
115 Check Shards Status After Follower Shutdown
116 [Documentation] Wait for node convergence and check status for all shards in OpenFlow application.
117 Wait Until Keyword Succeeds ${operation_timeout} 2s ClusterOpenFlow.Check OpenFlow Shards Status ${Active_Nodes}
119 Start Mininet Connect To Follower Node
120 [Documentation] Start mininet with connection to Follower Node1.
121 ${mininet_conn_id}= MininetKeywords.Start Mininet Single Controller ${TOOLS_SYSTEM_IP} ${ODL_SYSTEM_${Follower_Node_1}_IP}
122 Set Suite Variable ${mininet_conn_id}
124 Add Bulk Flow From Follower Node1
125 [Documentation] 1000 Flows added via Follower Node1 and verify it gets applied in all instances.
126 BulkomaticKeywords.Add Bulk Flow In Node ${Follower_Node_1} ${temp_json_config_add} ${operation_timeout}
128 Get Bulk Flows and Verify In Cluster Before Follower Restart
129 [Documentation] Initiate get operation and check flow count only across active cluster nodes
130 BulkomaticKeywords.Get Bulk Flow And Verify Count In Cluster ${Active_Nodes} ${temp_json_config_get} ${operation_timeout} ${flow_count_after_add}
132 Verify Flows In Switch Before Follower Restart
133 [Documentation] Verify flows are installed in switch before follower restart.
134 MininetKeywords.Verify Aggregate Flow From Mininet Session ${mininet_conn_id} ${switch_count} ${flow_count_per_switch} ${operation_timeout}
136 Restart Follower From Cluster Node
137 [Documentation] Restart Follower Node2.
138 ClusterKeywords.Start Multiple Controllers ${operation_timeout} ${Follower_Node_2}
140 Verify Data Recovery After Follower Restart
141 [Documentation] 1000 Flows preserved in all controller instances.
142 Wait Until Keyword Succeeds ${restart_timeout} 2s BulkomaticKeywords.Get Bulk Flow And Verify Count In Cluster ${original_cluster_list} ${temp_json_config_get} ${operation_timeout}
143 ... ${flow_count_after_add}
145 Verify Flows In Switch After Follower Restart
146 [Documentation] Verify flows are installed in switch after cluster restart.
147 MininetKeywords.Verify Aggregate Flow From Mininet Session ${mininet_conn_id} ${switch_count} ${flow_count_per_switch} ${operation_timeout}
149 Stop Mininet Connected To Follower Node
150 [Documentation] Stop mininet and exit connection.
151 MininetKeywords.Stop Mininet And Exit ${mininet_conn_id}
152 Utils.Clean Mininet System
154 Delete All Flows From Follower Node
155 [Documentation] 1000 Flows deleted via Follower Node1 and verify it gets applied in all instances.
156 BulkomaticKeywords.Delete Bulk Flow In Node ${Follower_Node_1} ${temp_json_config_del} ${operation_timeout}
158 Verify No Flows In Cluster After Follower Restart
159 [Documentation] Verify flow count is 0 across cluster nodes.
160 BulkomaticKeywords.Get Bulk Flow And Verify Count In Cluster ${original_cluster_list} ${temp_json_config_get} ${operation_timeout} ${flow_count_after_del}