2 Documentation Switch connections and cluster are restarted.
4 Library RequestsLibrary
6 Resource ../../../libraries/SetupUtils.robot
7 Resource ../../../libraries/ClusterManagement.robot
8 Resource ../../../libraries/ClusterOpenFlow.robot
9 Resource ../../../libraries/TemplatedRequests.robot
10 Resource ../../../libraries/MininetKeywords.robot
11 Resource ../../../libraries/Utils.robot
12 Resource ../../../libraries/OvsManager.robot
13 Resource ../../../variables/Variables.robot
14 Resource ../../../variables/openflowplugin/Variables.robot
16 Suite Setup Initialization Phase
17 Suite Teardown Final Phase
18 Test Setup SetupUtils.Setup_Test_With_Logging_And_Without_Fast_Failing
24 ${VAR_DIR} ${CURDIR}/../../../variables/openflowplugin
29 [Documentation] Add ${ITER} groups 1&2 and flows in every switch.
30 Add Groups And Flows On Member ${ITER}
32 Start Mininet Multiple Connections
33 [Documentation] Start mininet linear with connection to all cluster instances.
34 ${cluster_index_list}= ClusterManagement.List All Indices
35 ${mininet_conn_id}= MininetKeywords.Start Mininet Multiple Controllers
36 ... ${TOOLS_SYSTEM_IP}
37 ... ${cluster_index_list}
38 ... --topo linear,${SWITCHES}
39 BuiltIn.Set Suite Variable ${cluster_index_list}
40 BuiltIn.Set Suite Variable ${mininet_conn_id}
41 BuiltIn.Wait Until Keyword Succeeds
44 ... OVSDB.Check OVS OpenFlow Connections
45 ... ${TOOLS_SYSTEM_IP}
49 [Documentation] Check Linear Topology.
50 BuiltIn.Wait Until Keyword Succeeds 30s 1s ClusterOpenFlow.Check Linear Topology On Member ${SWITCHES}
52 Check Stats Are Not Frozen
53 [Documentation] Check that duration flow stat is increasing.
54 BuiltIn.Wait Until Keyword Succeeds 30s 1s Check Flow Stats Are Not Frozen
56 Check Flows In Operational DS
57 [Documentation] Check Flows in operational DS.
58 BuiltIn.Wait Until Keyword Succeeds 10s 1s ClusterOpenFlow.Check Number Of Flows On Member ${all_flows}
60 Check Groups In Operational DS
61 [Documentation] Check Groups in operational DS.
62 BuiltIn.Wait Until Keyword Succeeds
65 ... ClusterOpenFlow.Check Number Of Groups On Member
69 [Documentation] Check Flows in switch.
70 MininetKeywords.Check Flows In Mininet ${mininet_conn_id} ${all_flows}
72 Check Entity Owner Status And Find Owner and Successor Before Fail
73 [Documentation] Check Entity Owner Status and identify owner and successor for first switch s1.
74 ${original_owner} ${original_successor_list}= ClusterOpenFlow.Get OpenFlow Entity Owner Status For One Device
77 BuiltIn.Set Suite Variable ${original_owner}
78 BuiltIn.Set Suite Variable ${new_cluster_list} ${original_successor_list}
80 Disconnect Mininet From Owner
81 [Documentation] Disconnect mininet from the owner
82 ${original_owner_list}= BuiltIn.Create List ${original_owner}
83 MininetKeywords.Disconnect Cluster Mininet break ${original_owner_list}
84 BuiltIn.Set Suite Variable ${original_owner_list}
86 Check Entity Owner Status And Find Owner and Successor After Fail
87 [Documentation] Check Entity Owner Status and identify owner and successor for first switch s1.
88 ${new_owner} ${new_successor_list}= BuiltIn.Wait Until Keyword Succeeds
91 ... ClusterOpenFlow.Get OpenFlow Entity Owner Status For One Device
94 ... ${new_cluster_list}
96 ${owner_list}= BuiltIn.Create List ${original_owner} ${new_owner}
97 BuiltIn.Set Suite Variable ${owner_list}
98 BuiltIn.Set Suite Variable ${new_owner}
99 BuiltIn.Set Suite Variable ${new_successor_list}
101 Check Switch Moves To New Master
102 [Documentation] Check switch s1 is connected to new Master.
103 ${new_master}= BuiltIn.Set Variable ${ODL_SYSTEM_${new_owner}_IP}
104 BuiltIn.Wait Until Keyword Succeeds
107 ... OvsManager.Should Be Master
110 ... update_data=${True}
112 Check Linear Topology After Disconnect
113 [Documentation] Check Linear Topology.
114 BuiltIn.Wait Until Keyword Succeeds 30s 1s ClusterOpenFlow.Check Linear Topology On Member ${SWITCHES}
116 Check Stats Are Not Frozen After Disconnect
117 [Documentation] Check that duration flow stat is increasing
118 BuiltIn.Wait Until Keyword Succeeds 30s 1s Check Flow Stats Are Not Frozen
120 Remove Flows And Groups After Mininet Is Disconnected
121 [Documentation] Remove 1 group 1&2 and 1 flow in every switch.
122 Remove Single Group And Flow On Member
124 Check Flows In Operational DS After Mininet Is Disconnected
125 [Documentation] Check Flows in Operational DS.
126 BuiltIn.Wait Until Keyword Succeeds
129 ... ClusterOpenFlow.Check Number Of Flows On Member
132 Check Groups In Operational DS After Mininet Is Disconnected
133 [Documentation] Check Groups in Operational DS.
134 BuiltIn.Wait Until Keyword Succeeds
137 ... ClusterOpenFlow.Check Number Of Groups On Member
140 Check Flows In Switch After Mininet Is Disconnected
141 [Documentation] Check Flows in switch.
142 MininetKeywords.Check Flows In Mininet ${mininet_conn_id} ${less_flows}
144 Reconnect Mininet To Owner
145 [Documentation] Reconnect mininet to switch 1 owner.
146 MininetKeywords.Disconnect Cluster Mininet restore ${original_owner_list}
148 Check Entity Owner Status And Find Owner and Successor After Reconnect
149 [Documentation] Check Entity Owner Status and identify owner and successor for first switch s1.
150 BuiltIn.Wait Until Keyword Succeeds
153 ... ClusterOpenFlow.Get OpenFlow Entity Owner Status For One Device
157 Add Flows And Groups After Owner Reconnect
158 [Documentation] Add 1 group type 1&2 and 1 flow in every switch.
159 Add Single Group And Flow On Member
161 Check Stats Are Not Frozen After Owner Reconnect
162 [Documentation] Check that duration flow stat is increasing.
163 BuiltIn.Wait Until Keyword Succeeds 30s 1s Check Flow Stats Are Not Frozen
165 Check Flows After Owner Reconnect In Operational DS
166 [Documentation] Check Flows in Operational DS.
167 BuiltIn.Wait Until Keyword Succeeds 30s 1s ClusterOpenFlow.Check Number Of Flows On Member ${all_flows}
169 Check Groups After Owner Reconnect In Operational DS
170 [Documentation] Check Groups in Operational DS.
171 BuiltIn.Wait Until Keyword Succeeds
174 ... ClusterOpenFlow.Check Number Of Groups On Member
177 Check Flows After Owner Reconnect In Switch
178 [Documentation] Check Flows in switch.
179 MininetKeywords.Check Flows In Mininet ${mininet_conn_id} ${all_flows}
181 Check Switches Generate Slave Connection
182 [Documentation] Check switches are connected to new Slave.
183 ${original_master}= BuiltIn.Set Variable ${ODL_SYSTEM_${original_owner}_IP}
184 FOR ${switch} IN RANGE 1 ${switches+1}
185 BuiltIn.Wait Until Keyword Succeeds
188 ... OvsManager.Should Be Slave
190 ... ${original_master}
191 ... update_data=${True}
194 Disconnect Mininet From Successor
195 [Documentation] Disconnect mininet from the Successor.
196 MininetKeywords.Disconnect Cluster Mininet break ${new_successor_list}
198 Check Entity Owner Status And Find New Owner and Successor After Disconnect
199 [Documentation] Check Entity Owner Status and identify owner and successor for first switch s1.
200 ${current_owner} ${current_successor_list}= BuiltIn.Wait Until Keyword Succeeds
203 ... ClusterOpenFlow.Get OpenFlow Entity Owner Status For One Device
208 BuiltIn.Set Suite Variable ${current_owner}
209 BuiltIn.Set Suite Variable ${current_successor_list}
211 Disconnect Mininet From Current Owner
212 [Documentation] Disconnect mininet from the owner
213 ${current_owner_list}= BuiltIn.Create List ${current_owner}
214 MininetKeywords.Disconnect Cluster Mininet break ${current_owner_list}
216 Check Entity Owner Status And Find Current Owner and Successor After Disconnect
217 [Documentation] Check Entity Owner Status and identify owner and successor for first switch s1.
218 ${current_new_owner} ${current_new_successor_list}= BuiltIn.Wait Until Keyword Succeeds
221 ... ClusterOpenFlow.Get OpenFlow Entity Owner Status For One Device
224 ... ${original_owner_list}
226 BuiltIn.Set Suite Variable ${current_new_owner}
227 BuiltIn.Set Suite Variable ${current_new_successor_list}
229 Check Switch Moves To Current Master
230 [Documentation] Check switch s1 is connected to original Master.
231 ${current_new_master}= BuiltIn.Set Variable ${ODL_SYSTEM_${current_new_owner}_IP}
232 BuiltIn.Wait Until Keyword Succeeds
235 ... OvsManager.Should Be Master
237 ... ${current_new_master}
238 ... update_data=${True}
239 BuiltIn.Should Be Equal ${current_new_owner} ${original_owner}
241 Check Linear Topology After Owner Disconnect
242 [Documentation] Check Linear Topology.
243 BuiltIn.Wait Until Keyword Succeeds 30s 1s ClusterOpenFlow.Check Linear Topology On Member ${SWITCHES}
245 Check Stats Are Not Frozen After Owner Disconnect
246 [Documentation] Check that duration flow stat is increasing.
247 BuiltIn.Wait Until Keyword Succeeds 30s 1s Check Flow Stats Are Not Frozen
249 Remove Flows And Groups After Owner Disconnected
250 [Documentation] Remove 1 group 1&2 and 1 flow in every switch.
251 Remove Single Group And Flow On Member
253 Check Flows In Operational DS After Owner Disconnected
254 [Documentation] Check Flows in Operational DS.
255 BuiltIn.Wait Until Keyword Succeeds
258 ... ClusterOpenFlow.Check Number Of Flows On Member
261 Check Groups In Operational DS After Owner Disconnected
262 [Documentation] Check Groups in Operational DS.
263 BuiltIn.Wait Until Keyword Succeeds
266 ... ClusterOpenFlow.Check Number Of Groups On Member
269 Check Flows In Switch After Owner Disconnected
270 [Documentation] Check Flows in switch.
271 MininetKeywords.Check Flows In Mininet ${mininet_conn_id} ${less_flows}
273 Disconnect Mininet From Cluster
274 [Documentation] Disconnect Mininet from Cluster.
275 MininetKeywords.Disconnect Cluster Mininet break ${original_owner_list}
277 Check No Switches After Disconnect
278 [Documentation] Check no switches in topology.
279 BuiltIn.Wait Until Keyword Succeeds 30s 1s ClusterOpenFlow.Check No Switches On Member ${SWITCHES}
281 Check Switch Is Not Connected
282 [Documentation] Check switch s1 is not connected to any controller.
283 FOR ${index} IN @{cluster_index_list}
284 BuiltIn.Wait Until Keyword Succeeds
287 ... OvsManager.Should Be Disconnected
289 ... ${ODL_SYSTEM_${index}_IP}
290 ... update_data=${True}
293 Reconnect Mininet To Cluster
294 [Documentation] Reconnect mininet to cluster by removing Iptables drop rules that were used to disconnect
295 MininetKeywords.Disconnect Cluster Mininet restore
297 Check Linear Topology After Mininet Reconnects
298 [Documentation] Check Linear Topology.
299 BuiltIn.Wait Until Keyword Succeeds 10s 1s ClusterOpenFlow.Check Linear Topology On Member ${SWITCHES}
301 Add Flows And Groups After Mininet Reconnects
302 [Documentation] Add 1 group type 1&2 and 1 flow in every switch.
303 Add Single Group And Flow On Member
305 Check Flows In Operational DS After Mininet Reconnects
306 [Documentation] Check Flows in Operational DS.
307 BuiltIn.Wait Until Keyword Succeeds 30s 1s ClusterOpenFlow.Check Number Of Flows On Member ${all_flows}
309 Check Groups In Operational DS After Mininet Reconnects
310 [Documentation] Check Groups in Operational DS.
311 BuiltIn.Wait Until Keyword Succeeds
314 ... ClusterOpenFlow.Check Number Of Groups On Member
317 Check Flows In Switch After Mininet Reconnects
318 [Documentation] Check Flows in switch.
319 MininetKeywords.Check Flows In Mininet ${mininet_conn_id} ${all_flows}
321 Check Entity Owner Status And Find Owner and Successor Before Owner Stop
322 [Documentation] Check Entity Owner Status and identify owner and successor for first switch s1.
323 ${original_owner} ${original_successor_list}= ClusterOpenFlow.Get OpenFlow Entity Owner Status For One Device
326 ${original_successor}= Collections.Get From List ${original_successor_list} 0
327 BuiltIn.Set Suite Variable ${original_owner}
328 BuiltIn.Set Suite Variable ${original_successor_list}
329 BuiltIn.Set Suite Variable ${original_successor}
331 Check Switch Generates Slave Connection Before Owner Stop
332 [Documentation] Check switch s1 is connected to Slave.
333 ${original_slave}= BuiltIn.Set Variable ${ODL_SYSTEM_${original_successor}_IP}
334 BuiltIn.Wait Until Keyword Succeeds
337 ... OvsManager.Should Be Slave
339 ... ${original_slave}
340 ... update_data=${True}
342 Check Shards Status Before Owner Stop
343 [Documentation] Check Status for all shards in OpenFlow application.
344 ClusterOpenFlow.Check OpenFlow Shards Status
347 [Documentation] Stop Owner Instance and verify it is shutdown
348 ClusterManagement.Stop Single Member ${original_owner}
349 BuiltIn.Set Suite Variable ${new_cluster_list} ${original_successor_list}
351 Check Shards Status After Stop
352 [Documentation] Check Status for all shards in OpenFlow application.
353 ClusterOpenFlow.Check OpenFlow Shards Status After Cluster Event ${new_cluster_list}
355 Check Entity Owner Status And Find Owner and Successor After Stop
356 [Documentation] Check Entity Owner Status and identify owner and successor.
357 ${new_owner} ${new_successor_list}= ClusterOpenFlow.Get OpenFlow Entity Owner Status For One Device
359 ... ${original_successor}
360 ... ${new_cluster_list}
362 BuiltIn.Set Suite Variable ${new_owner}
364 Check Stats Are Not Frozen After Owner Stop
365 [Documentation] Check that duration flow stat is increasing in new owner.
366 BuiltIn.Wait Until Keyword Succeeds 30s 1s Check Flow Stats Are Not Frozen ${new_owner}
368 Remove Configuration In Owner and Verify After Owner Stop
369 [Documentation] Remove 1 group 1&2 and 1 flow in every switch.
370 Remove Single Group And Flow On Member ${new_owner}
372 Check Flows After Owner Stop In Operational DS
373 [Documentation] Check Flows in Operational DS in new owner.
374 BuiltIn.Wait Until Keyword Succeeds
377 ... ClusterOpenFlow.Check Number Of Flows On Member
381 Check Groups After Owner Stop In Operational DS
382 [Documentation] Check Groups in Operational DS in new owner.
383 BuiltIn.Wait Until Keyword Succeeds
386 ... ClusterOpenFlow.Check Number Of Groups On Member
390 Check Flows In Switch After Owner Stop
391 [Documentation] Check Flows in switch.
392 MininetKeywords.Check Flows In Mininet ${mininet_conn_id} ${less_flows}
394 Start Old Owner Instance
395 [Documentation] Start old Owner Instance and verify it is up.
396 ClusterManagement.Start Single Member ${original_owner}
398 Check Entity Owner Status And Find Owner and Successor After Start Owner
399 [Documentation] Check Entity Owner Status and identify owner and successor for first switch s1.
400 ${owner} ${successor_list}= BuiltIn.Wait Until Keyword Succeeds
403 ... ClusterOpenFlow.Get OpenFlow Entity Owner Status For One Device
407 Check Linear Topology After Owner Restart
408 [Documentation] Check Linear Topology.
409 BuiltIn.Wait Until Keyword Succeeds 10s 1s ClusterOpenFlow.Check Linear Topology On Member ${SWITCHES}
411 Add Configuration In Owner and Verify After Owner Restart
412 [Documentation] Add 1 group type 1&2 and 1 flow in every switch.
413 Add Single Group And Flow On Member ${new_owner}
415 Check Stats Are Not Frozen After Owner Restart
416 [Documentation] Check that duration flow stat is increasing.
418 BuiltIn.Wait Until Keyword Succeeds 30s 1s Check Flow Stats Are Not Frozen
420 Check Flows In Operational DS After Owner Restart
421 [Documentation] Check Flows in Operational DS.
422 BuiltIn.Wait Until Keyword Succeeds 30s 1s ClusterOpenFlow.Check Number Of Flows On Member ${all_flows}
424 Check Groups In Operational DS After Owner Restart
425 [Documentation] Check Groups in Operational DS.
426 BuiltIn.Wait Until Keyword Succeeds
429 ... ClusterOpenFlow.Check Number Of Groups On Member
432 Check Flows In Switch After Owner Restart
433 [Documentation] Check Flows in switch.
434 MininetKeywords.Check Flows In Mininet ${mininet_conn_id} ${all_flows}
437 [Documentation] Stop and Start cluster.
438 # Try to stop contoller, if stop does not work or takes too long, kill controller.
439 ${status} ${result}= BuiltIn.Run Keyword And Ignore Error ClusterManagement.Stop_Members_From_List_Or_All
440 IF '${status}' != 'PASS'
441 ClusterManagement.Kill_Members_From_List_Or_All
443 ClusterManagement.Start_Members_From_List_Or_All wait_for_sync=False
445 Check Linear Topology After Controller Restarts
446 [Documentation] Check Linear Topology
447 BuiltIn.Wait Until Keyword Succeeds 300s 2s ClusterOpenFlow.Check Linear Topology On Member ${SWITCHES}
449 Check Stats Are Not Frozen After Cluster Restart
450 [Documentation] Check that duration flow stat is increasing.
451 BuiltIn.Wait Until Keyword Succeeds 30s 1s Check Flow Stats Are Not Frozen
453 Check Flows In Operational DS After Controller Restarts
454 [Documentation] Check Flows in Operational DS.
455 BuiltIn.Wait Until Keyword Succeeds 30s 1s ClusterOpenFlow.Check Number Of Flows On Member ${all_flows}
457 Check Groups In Operational DS After Controller Restarts
458 [Documentation] Check Groups in Operational DS.
459 BuiltIn.Wait Until Keyword Succeeds
462 ... ClusterOpenFlow.Check Number Of Groups On Member
465 Check Flows In Switch After Controller Restarts
466 [Documentation] Check Flows in switch.
467 MininetKeywords.Check Flows In Mininet ${mininet_conn_id} ${all_flows}
470 [Documentation] Stop Mininet.
471 MininetKeywords.Stop Mininet And Exit
474 [Documentation] Check no switches in topology.
475 BuiltIn.Wait Until Keyword Succeeds 5s 1s ClusterOpenFlow.Check No Switches On Member ${SWITCHES}
480 [Documentation] Create controller session and set variables.
481 SetupUtils.Setup_Utils_For_Setup_And_Teardown
482 RequestsLibrary.Create Session session http://${ODL_SYSTEM_IP}:${RESTCONFPORT} auth=${AUTH}
483 ${switches}= Convert To Integer ${SWITCHES}
484 ${iter}= Convert To Integer ${ITER}
485 ${all_groups}= BuiltIn.Evaluate ${switches} * ${iter} * 2
486 ${less_groups}= BuiltIn.Evaluate ${all_groups} - ${switches} * 2
487 # Stale flows/groups feature enabled in Boron onwards.
488 ${all_flows}= BuiltIn.Evaluate ${switches} * ${iter+1}
489 ${less_flows}= BuiltIn.Evaluate ${all_flows} - ${switches}
490 # Stale flows/groups feature enabled in Boron onwards.
491 BuiltIn.Set Suite Variable ${switches}
492 BuiltIn.Set Suite Variable ${iter}
493 BuiltIn.Set Suite Variable ${all_groups}
494 BuiltIn.Set Suite Variable ${less_groups}
495 BuiltIn.Set Suite Variable ${all_flows}
496 BuiltIn.Set Suite Variable ${less_flows}
497 BuiltIn.Set Suite Variable ${no_flows} ${SWITCHES}
500 [Documentation] Delete all sessions.
501 ${command}= BuiltIn.Set Variable sudo iptables -v -F
502 BuiltIn.Run Keyword And Ignore Error ClusterManagement.Run_Bash_Command_On_List_Or_All ${command}
503 BuiltIn.Run Keyword And Ignore Error RequestsLibrary.DELETE On Session session url=${RFC8040_NODES_API}
504 RequestsLibrary.Delete All Sessions
506 Add Groups And Flows On Member
507 [Documentation] Add ${ITER} groups type 1 & 2 and flows in every switch.
508 [Arguments] ${iter}=1 ${member_index}=1
509 ${session}= Resolve_Http_Session_For_Member member_index=${member_index}
510 FOR ${switch} IN RANGE 1 ${switches+1}
511 &{mapping}= BuiltIn.Create_Dictionary NODE=openflow:${switch}
512 TemplatedRequests.Post As Json Templated
513 ... folder=${VAR_DIR}/add-group-1
514 ... mapping=${mapping}
515 ... session=${session}
516 ... iterations=${iter}
517 TemplatedRequests.Post As Json Templated
518 ... folder=${VAR_DIR}/add-group-2
519 ... mapping=${mapping}
520 ... session=${session}
521 ... iterations=${iter}
522 TemplatedRequests.Post As Json Templated
523 ... folder=${VAR_DIR}/add-flow
524 ... mapping=${mapping}
525 ... session=${session}
526 ... iterations=${iter}
529 Add Single Group And Flow On Member
530 [Documentation] Add 1 group 1&2 and 1 flow in every switch.
531 [Arguments] ${member_index}=1
532 Add Groups And Flows On Member 1 ${member_index}
534 Remove Single Group And Flow On Member
535 [Documentation] Remove 1 group 1&2 and 1 flow in every switch.
536 [Arguments] ${member_index}=1
537 ${session}= Resolve_Http_Session_For_Member member_index=${member_index}
538 FOR ${switch} IN RANGE 1 ${switches+1}
539 RequestsLibrary.DELETE On Session
541 ... url=${RFC8040_NODES_API}/node=openflow%3A${switch}/flow-node-inventory:table=0/flow=1
542 RequestsLibrary.DELETE On Session
544 ... url=${RFC8040_NODES_API}/node=openflow%3A${switch}/flow-node-inventory:group=1
545 RequestsLibrary.DELETE On Session
547 ... url=${RFC8040_NODES_API}/node=openflow%3A${switch}/flow-node-inventory:group=1000
550 Check Flow Stats Are Not Frozen
551 [Documentation] Verify flow stats are not frozen for flow 1 and switch 1.
552 [Arguments] ${member_index}=1 ${period_in_seconds}=5
553 ${duration_1}= Extract Flow Duration ${member_index}
554 ${duration_1}= Builtin.Convert To Integer ${duration_1}
555 BuiltIn.Sleep ${period_in_seconds}
556 ${duration_2}= Extract Flow Duration ${member_index}
557 ${duration_2}= Builtin.Convert To Integer ${duration_2}
558 Should Not Be Equal As Integers ${duration_1} ${duration_2}
560 Extract Flow Duration
561 [Documentation] Extract duration for flow 1 in switch 1.
562 [Arguments] ${member_index}
563 ${session}= Resolve_Http_Session_For_Member member_index=${member_index}
564 ${resp}= RequestsLibrary.GET On Session
566 ... url=${RFC8040_NODES_API}/node=openflow%3A1/flow-node-inventory:table=0/flow=1?content=nonconfig
567 ... headers=${headers}
569 ${json_resp}= Utils.Json Parse From String ${resp.content}
570 ${flow_list}= Collections.Get_From_Dictionary ${json_resp} flow-node-inventory:flow
571 ${flow_stats}= Collections.Get_From_Dictionary
573 ... opendaylight-flow-statistics:flow-statistics
574 ${duration}= Collections.Get_From_Dictionary ${flow_stats}[duration] second