2 Documentation Test suite for Cluster HA with Bulk Flows - Data consistency after cluster restart, leader restart and follower restart with one switch connected
3 Suite Setup SetupUtils.Setup_Utils_For_Setup_And_Teardown
4 Suite Teardown Delete All Sessions
5 Test Setup SetupUtils.Setup_Test_With_Logging_And_Without_Fast_Failing
6 Resource ../../../libraries/SetupUtils.robot
7 Resource ../../../libraries/BulkomaticKeywords.robot
8 Resource ../../../libraries/MininetKeywords.robot
9 Resource ../../../libraries/ClusterManagement.robot
10 Resource ../../../libraries/ClusterOpenFlow.robot
11 Resource ../../../libraries/Utils.robot
12 Variables ../../../variables/Variables.py
15 ${operation_timeout} 100s
16 ${restart_timeout} 350s
17 ${flow_count_per_switch} 1000
19 ${flow_count_after_add} 1000
20 ${flow_count_after_del} 0
21 ${orig_json_config_add} sal_add_bulk_flow_config.json
22 ${orig_json_config_get} sal_get_bulk_flow_config.json
23 ${orig_json_config_del} sal_del_bulk_flow_config.json
26 Check Shards Status And Initialize Variables
27 [Documentation] Check Status for all shards in OpenFlow application.
28 ClusterOpenFlow.Check OpenFlow Shards Status
29 ${temp_json_config_add} BulkomaticKeywords.Set DPN And Flow Count In Json Add ${orig_json_config_add} ${switch_count} ${flow_count_per_switch}
30 ${temp_json_config_get} BulkomaticKeywords.Set DPN And Flow Count In Json Get ${orig_json_config_get} ${switch_count} ${flow_count_after_add}
31 ${temp_json_config_del} BulkomaticKeywords.Set DPN And Flow Count In Json Del ${orig_json_config_del} ${switch_count} ${flow_count_per_switch}
32 Set Suite Variable ${temp_json_config_add}
33 Set Suite Variable ${temp_json_config_get}
34 Set Suite Variable ${temp_json_config_del}
36 Get Inventory Follower Before Cluster Restart
37 [Documentation] Find a follower in the inventory config shard
38 ${inventory_leader} ${inventory_followers} ClusterOpenFlow.Get InventoryConfig Shard Status
39 ${Follower_Node_1}= Get From List ${Inventory_Followers} 0
40 Set Suite Variable ${Follower_Node_1}
42 Start Mininet Connect To Follower Node1
43 [Documentation] Start mininet with connection to Follower Node1.
44 ${mininet_conn_id}= MininetKeywords.Start Mininet Single Controller ${TOOLS_SYSTEM_IP} ${ODL_SYSTEM_${Follower_Node_1}_IP}
45 Set Suite Variable ${mininet_conn_id}
47 Add Bulk Flow From Follower
48 [Documentation] 1000 Flows added via Follower Node1 and verify it gets applied in all instances.
49 BulkomaticKeywords.Add Bulk Flow In Node ${temp_json_config_add} ${Follower_Node_1} ${operation_timeout}
51 Get Bulk Flows and Verify In Cluster
52 [Documentation] Initiate get operation and check flow count across cluster nodes
53 BulkomaticKeywords.Get Bulk Flow And Verify Count In Cluster ${temp_json_config_get} ${operation_timeout} ${flow_count_after_add}
55 Verify Flows In Switch Before Cluster Restart
56 [Documentation] Verify flows are installed in switch before cluster restart.
57 MininetKeywords.Verify Aggregate Flow From Mininet Session ${mininet_conn_id} ${flow_count_after_add} ${operation_timeout}
59 Kill All Cluster Nodes
60 [Documentation] Kill All Nodes.
61 ClusterManagement.Kill Members From List Or All
63 Stop Mininet Connected To Follower Node1 and Exit
64 [Documentation] Stop mininet and exit connection.
65 MininetKeywords.Stop Mininet And Exit ${mininet_conn_id}
66 Utils.Clean Mininet System
68 Restart All Cluster Nodes
69 [Documentation] Restart all cluster nodes.
70 ClusterManagement.Start Members From List Or All
72 Verify Data Recovery After Cluster Restart
73 [Documentation] 1000 Flows preserved in all controller instances.
74 Wait Until Keyword Succeeds ${restart_timeout} 2s BulkomaticKeywords.Get Bulk Flow And Verify Count In Cluster ${temp_json_config_get} ${operation_timeout} ${flow_count_after_add}
76 Start Mininet Again Connect To Follower Node1
77 [Documentation] Start mininet with connection to follower node1.
78 ${mininet_conn_id}= MininetKeywords.Start Mininet Single Controller ${TOOLS_SYSTEM_IP} ${ODL_SYSTEM_${Follower_Node_1}_IP}
79 Set Suite Variable ${mininet_conn_id}
81 Verify Flows In Switch After Cluster Restart
82 [Documentation] Verify flows are installed in switch after cluster restart.
83 MininetKeywords.Verify Aggregate Flow From Mininet Session ${mininet_conn_id} ${flow_count_after_add} ${operation_timeout}
85 Stop Mininet Connected To Follower Node1
86 [Documentation] Stop mininet and exit connection.
87 MininetKeywords.Stop Mininet And Exit ${mininet_conn_id}
88 Utils.Clean Mininet System
90 Delete All Flows From Follower Node1
91 [Documentation] 1000 Flows deleted via Follower Node1 and verify it gets applied in all instances.
92 BulkomaticKeywords.Delete Bulk Flow In Node ${temp_json_config_del} ${Follower_Node_1} ${operation_timeout}
94 Verify No Flows In Cluster
95 [Documentation] Verify flow count is 0 across cluster nodes.
96 BulkomaticKeywords.Get Bulk Flow And Verify Count In Cluster ${temp_json_config_get} ${operation_timeout} ${flow_count_after_del}
98 Get Inventory Leader Before Leader Restart
99 [Documentation] Find leader in the inventory config shard
100 ${inventory_leader} ${inventory_followers} ClusterOpenFlow.Get InventoryConfig Shard Status
101 Set Suite Variable ${Inventory_Leader}
103 Start Mininet Connect To Leader
104 [Documentation] Start mininet with connection to Leader Node.
105 ${mininet_conn_id}= MininetKeywords.Start Mininet Single Controller ${TOOLS_SYSTEM_IP} ${ODL_SYSTEM_${Inventory_Leader}_IP}
106 Set Suite Variable ${mininet_conn_id}
108 Add Bulk Flow From Leader
109 [Documentation] 1000 Flows added via Follower Node1 and verify it gets applied in all instances.
110 BulkomaticKeywords.Add Bulk Flow In Node ${temp_json_config_add} ${Inventory_Leader} ${operation_timeout}
112 Get Bulk Flows and Verify In Cluster Before Leader Restart
113 [Documentation] Initiate get operation and check flow count across cluster nodes
114 BulkomaticKeywords.Get Bulk Flow And Verify Count In Cluster ${temp_json_config_get} ${operation_timeout} ${flow_count_after_add}
116 Verify Flows In Switch Before Leader Restart
117 [Documentation] Verify flows are installed in switch before leader restart.
118 MininetKeywords.Verify Aggregate Flow From Mininet Session ${mininet_conn_id} ${flow_count_after_add} ${operation_timeout}
120 Kill Leader From Cluster Node
121 [Documentation] Kill Leader Node.
122 ClusterManagement.Kill Single Member ${Inventory_Leader}
124 Stop Mininet Connected To Leader Node
125 [Documentation] Stop mininet and exit connection.
126 MininetKeywords.Stop Mininet And Exit ${mininet_conn_id}
127 Utils.Clean Mininet System
129 Restart Leader from Cluster Node
130 [Documentation] Start Leader Node Up.
131 ClusterManagement.Start Single Member ${Inventory_Leader}
133 Verify Data Recovery After Leader Restart
134 [Documentation] 1000 Flows preserved in all controller instances.
135 Wait Until Keyword Succeeds ${restart_timeout} 2s BulkomaticKeywords.Get Bulk Flow And Verify Count In Cluster ${temp_json_config_get} ${operation_timeout} ${flow_count_after_add}
137 Check No Network Operational Information After Leader Restart
138 [Documentation] Check device is not in operational inventory or topology in all cluster instances.
139 ClusterOpenFlow.Check No OpenFlow Network Operational Information
141 Start Mininet Again Connect To Leader
142 [Documentation] Start mininet with connection to Leader Node.
143 ${mininet_conn_id}= MininetKeywords.Start Mininet Single Controller ${TOOLS_SYSTEM_IP} ${ODL_SYSTEM_${Inventory_Leader}_IP}
144 Set Suite Variable ${mininet_conn_id}
146 Verify Flows In Switch After Leader Restart
147 [Documentation] Verify flows are installed in switch after leader restart.
148 MininetKeywords.Verify Aggregate Flow From Mininet Session ${mininet_conn_id} ${flow_count_after_add} ${operation_timeout}
150 Stop Mininet Connected To Leader Node After Leader Restart
151 [Documentation] Stop mininet and exit connection.
152 MininetKeywords.Stop Mininet And Exit ${mininet_conn_id}
153 Utils.Clean Mininet System
155 Delete All Flows From Leader Node
156 [Documentation] 1000 Flows deleted via Leader Node and verify it gets applied in all instances.
157 BulkomaticKeywords.Delete Bulk Flow In Node ${temp_json_config_del} ${Inventory_Leader} ${operation_timeout}
159 Verify No Flows In Cluster After Leader Restart
160 [Documentation] Verify flow count is 0 across cluster nodes.
161 BulkomaticKeywords.Get Bulk Flow And Verify Count In Cluster ${temp_json_config_get} ${operation_timeout} ${flow_count_after_del}
163 Get Inventory Follower Before follower Restart
164 [Documentation] Find follower in the inventory config shard
165 ${inventory_leader} ${inventory_followers} ClusterOpenFlow.Get InventoryConfig Shard Status
166 ${Follower_Node_2}= Get From List ${Inventory_Followers} 1
167 Set Suite Variable ${Follower_Node_2}
169 Start Mininet Connect To Follower Node2
170 [Documentation] Start mininet with connection to Follower Node2.
171 ${mininet_conn_id}= MininetKeywords.Start Mininet Single Controller ${TOOLS_SYSTEM_IP} ${ODL_SYSTEM_${Follower_Node_2}_IP}
172 Set Suite Variable ${mininet_conn_id}
174 Add Bulk Flow From Follower Node2
175 [Documentation] 1000 Flows added via Follower Node2 and verify it gets applied in all instances.
176 BulkomaticKeywords.Add Bulk Flow In Node ${temp_json_config_add} ${Follower_Node_2} ${operation_timeout}
178 Get Bulk Flows and Verify In Cluster Before Follower Restart
179 [Documentation] Initiate get operation and check flow count across cluster nodes.
180 BulkomaticKeywords.Get Bulk Flow And Verify Count In Cluster ${temp_json_config_get} ${operation_timeout} ${flow_count_after_add}
182 Verify Flows In Switch Before Follower Restart
183 [Documentation] Verify flows are installed in switch before follower restart.
184 MininetKeywords.Verify Aggregate Flow From Mininet Session ${mininet_conn_id} ${flow_count_after_add} ${operation_timeout}
187 [Documentation] Kill Follower Node2.
188 ClusterManagement.Kill Single Member ${Follower_Node_2}
190 Stop Mininet Connected To Follower Node2 and Exit
191 [Documentation] Stop mininet and exit connection.
192 MininetKeywords.Stop Mininet And Exit ${mininet_conn_id}
193 Utils.Clean Mininet System
195 Restart Follower Node2
196 [Documentation] Start Follower Node2 Up.
197 ClusterManagement.Start Single Member ${Follower_Node_2}
199 Verify Data Recovery After Follower Node2 Restart
200 [Documentation] 1000 Flows preserved in all controller instances.
201 Wait Until Keyword Succeeds ${restart_timeout} 2s BulkomaticKeywords.Get Bulk Flow And Verify Count In Cluster ${temp_json_config_get} ${operation_timeout} ${flow_count_after_add}
203 Check No Network Operational Information After Follower Node2 Restart
204 [Documentation] Check device is not in operational inventory or topology in all cluster instances.
205 ClusterOpenFlow.Check No OpenFlow Network Operational Information
207 Start Mininet Again Connect To Follower Node2
208 [Documentation] Start mininet with connection to follower node1.
209 ${mininet_conn_id}= MininetKeywords.Start Mininet Single Controller ${TOOLS_SYSTEM_IP} ${ODL_SYSTEM_${Follower_Node_2}_IP}
210 Set Suite Variable ${mininet_conn_id}
212 Verify Flows In Switch After Follower Node2 Restart
213 [Documentation] Verify flows are installed in switch after follower restart.
214 MininetKeywords.Verify Aggregate Flow From Mininet Session ${mininet_conn_id} ${flow_count_after_add} ${operation_timeout}
216 Stop Mininet Connected To Follower Node2
217 [Documentation] Stop mininet and exit connection.
218 MininetKeywords.Stop Mininet And Exit ${mininet_conn_id}
219 Utils.Clean Mininet System
221 Delete All Flows From Follower Node 2
222 [Documentation] 1000 Flows deleted via Leader Node and verify it gets applied in all instances.
223 BulkomaticKeywords.Delete Bulk Flow In Node ${temp_json_config_del} ${Follower_Node_2} ${operation_timeout}
225 Verify No Flows In Cluster After Follower Node2 Restart
226 BulkomaticKeywords.Get Bulk Flow And Verify Count In Cluster ${temp_json_config_get} ${operation_timeout} ${flow_count_after_del}