2 Documentation Test suite to check connectivity in L3 using routers.
5 Library OperatingSystem
6 Library RequestsLibrary
8 Resource ../../../libraries/Utils.robot
9 Resource ../../../libraries/OpenStackOperations.robot
10 Resource ../../../libraries/DevstackUtils.robot
11 Resource ../../../libraries/OVSDB.robot
12 Resource ../../../libraries/ClusterOvsdb.robot
13 Resource ../../../libraries/ClusterManagement.robot
14 Resource ../../../libraries/SetupUtils.robot
15 Resource ../../../variables/Variables.robot
17 Suite Setup OpenStackOperations.OpenStack Suite Setup
18 Suite Teardown OpenStackOperations.OpenStack Suite Teardown
19 Test Setup SetupUtils.Setup_Test_With_Logging_And_Without_Fast_Failing
20 Test Teardown OpenStackOperations.Get Test Teardown Debugs
24 ${SECURITY_GROUP} cl3_sg
25 @{NETWORKS} cl3_net_1 cl3_net_2
26 @{SUBNETS} cl3_sub_1 cl3_sub_2
27 @{ROUTERS} cl3_router_1 cl3_router_2 cl3_router_3
28 @{NET_1_VMS} cl3_net_1_vm_1 cl3_net_1_vm_2 cl3_net_1_vm_3
29 @{NET_2_VMS} cl3_net_2_vm_1 cl3_net_2_vm_2 cl3_net_2_vm_3
30 @{SUBNET_CIDRS} 36.0.0.0/24 37.0.0.0/24
31 @{GATEWAY_IPS} 36.0.0.1 37.0.0.1
32 @{ODL_1_AND_2_DOWN} ${1} ${2}
33 @{ODL_2_AND_3_DOWN} ${2} ${3}
38 Create All Controller Sessions
39 [Documentation] Create sessions for all three controllers.
40 ClusterManagement.ClusterManagement Setup
41 BuiltIn.Run Keyword And Ignore Error
42 ... ClusterManagement.Get Raft State Of Shard Of All Member Nodes
43 ... shard_name=default
46 Take Down Leader Of Default Shard
47 [Documentation] Stop the karaf on ODL cluster leader
48 BuiltIn.Run Keyword And Ignore Error
49 ... ClusterManagement.Get Raft State Of Shard Of All Member Nodes
50 ... shard_name=default
52 ${cluster_leader} ${followers} = ClusterManagement.Get Leader And Followers For Shard shard_type=config
53 BuiltIn.Set Suite Variable ${cluster_leader}
54 ${new_cluster_list} = ClusterManagement.Stop Single Member
56 ... msg=up: ODL1, ODL2, ODL3, down=none
57 BuiltIn.Set Suite Variable ${new_cluster_list}
58 BuiltIn.Run Keyword And Ignore Error
59 ... ClusterManagement.Get Raft State Of Shard Of All Member Nodes
60 ... shard_name=default
64 [Documentation] Create Network with neutron request.
65 FOR ${NetworkElement} IN @{NETWORKS}
66 OpenStackOperations.Create Network ${NetworkElement}
69 Create Subnets For net_1
70 [Documentation] Create Sub Nets for the Networks with neutron request.
71 OpenStackOperations.Create SubNet ${NETWORKS}[0] ${SUBNETS}[0] ${SUBNET_CIDRS}[0]
73 Create Subnets For net_2
74 [Documentation] Create Sub Nets for the Networks with neutron request.
75 OpenStackOperations.Create SubNet ${NETWORKS}[1] ${SUBNETS}[1] ${SUBNET_CIDRS}[1]
77 Bring Up Leader Of Default Shard
78 [Documentation] Bring up on cluster leader
79 ClusterManagement.Start Single Member ${cluster_leader} msg=up: ${new_cluster_list}, down: ${cluster_leader}
80 BuiltIn.Run Keyword And Ignore Error
81 ... ClusterManagement.Get Raft State Of Shard Of All Member Nodes
82 ... shard_name=default
85 Add Ssh Allow All Rule
86 [Documentation] Allow all TCP/UDP/ICMP packets for this suite
87 OpenStackOperations.Create Allow All SecurityGroup ${SECURITY_GROUP}
90 [Documentation] Stop the karaf in First Controller
91 BuiltIn.Run Keyword And Ignore Error
92 ... ClusterManagement.Get Raft State Of Shard Of All Member Nodes
93 ... shard_name=default
95 ClusterManagement.Stop Single Member 1 msg=up: ODL1, ODL2, ODL3, down=none
96 BuiltIn.Run Keyword And Ignore Error
97 ... ClusterManagement.Get Raft State Of Shard Of All Member Nodes
98 ... shard_name=default
101 Create Vm Instances For net_1
102 [Documentation] Create Vm instances using flavor and image names for a network.
103 OpenStackOperations.Create Vm Instance On Compute Node
106 ... ${OS_CMP1_HOSTNAME}
107 ... sg=${SECURITY_GROUP}
108 OpenStackOperations.Create Vm Instance On Compute Node
111 ... ${OS_CMP1_HOSTNAME}
112 ... sg=${SECURITY_GROUP}
113 OpenStackOperations.Create Vm Instance On Compute Node
116 ... ${OS_CMP2_HOSTNAME}
117 ... sg=${SECURITY_GROUP}
120 [Documentation] Bring up ODL1 again
121 BuiltIn.Run Keyword And Ignore Error
122 ... ClusterManagement.Get Raft State Of Shard Of All Member Nodes
123 ... shard_name=default
124 ... shard_type=config
125 ClusterManagement.Start Single Member 1 msg=up: ODL2, ODL3, down: ODL1
126 BuiltIn.Run Keyword And Ignore Error
127 ... ClusterManagement.Get Raft State Of Shard Of All Member Nodes
128 ... shard_name=default
129 ... shard_type=config
132 [Documentation] Stop the karaf in Second Controller
133 BuiltIn.Run Keyword And Ignore Error
134 ... ClusterManagement.Get Raft State Of Shard Of All Member Nodes
135 ... shard_name=default
136 ... shard_type=config
137 ClusterManagement.Stop Single Member 2 msg=up: ODL1, ODL2, ODL3, down=none
138 BuiltIn.Run Keyword And Ignore Error
139 ... ClusterManagement.Get Raft State Of Shard Of All Member Nodes
140 ... shard_name=default
141 ... shard_type=config
143 Create Vm Instances For net_2
144 [Documentation] Create Vm instances using flavor and image names for a network.
145 OpenStackOperations.Create Vm Instance On Compute Node
148 ... ${OS_CMP1_HOSTNAME}
149 ... sg=${SECURITY_GROUP}
150 OpenStackOperations.Create Vm Instance On Compute Node
153 ... ${OS_CMP2_HOSTNAME}
154 ... sg=${SECURITY_GROUP}
155 OpenStackOperations.Create Vm Instance On Compute Node
158 ... ${OS_CMP2_HOSTNAME}
159 ... sg=${SECURITY_GROUP}
161 Check Vm Instances Have Ip Address
162 @{NET_1_L3_VM_IPS} ${NET_1_DHCP_IP} = OpenStackOperations.Get VM IPs @{NET_1_VMS}
163 @{NET_2_L3_VM_IPS} ${NET_2_DHCP_IP} = OpenStackOperations.Get VM IPs @{NET_2_VMS}
164 BuiltIn.Set Suite Variable @{NET_1_L3_VM_IPS}
165 BuiltIn.Set Suite Variable @{NET_2_L3_VM_IPS}
166 BuiltIn.Should Not Contain ${NET_1_L3_VM_IPS} None
167 BuiltIn.Should Not Contain ${NET_2_L3_VM_IPS} None
168 BuiltIn.Should Not Contain ${NET_1_DHCP_IP} None
169 BuiltIn.Should Not Contain ${NET_2_DHCP_IP} None
170 [Teardown] BuiltIn.Run Keywords OpenStackOperations.Show Debugs @{NET_1_VMS} @{NET_2_VMS}
171 ... AND OpenStackOperations.Get Test Teardown Debugs
174 [Documentation] Bring up ODL2 again
175 BuiltIn.Run Keyword And Ignore Error
176 ... ClusterManagement.Get Raft State Of Shard Of All Member Nodes
177 ... shard_name=default
178 ... shard_type=config
179 ClusterManagement.Start Single Member 2 msg=up: ODL1, ODL3, down: ODL2
180 BuiltIn.Run Keyword And Ignore Error
181 ... ClusterManagement.Get Raft State Of Shard Of All Member Nodes
182 ... shard_name=default
183 ... shard_type=config
186 [Documentation] Stop the karaf in Third Controller
187 BuiltIn.Run Keyword And Ignore Error
188 ... ClusterManagement.Get Raft State Of Shard Of All Member Nodes
189 ... shard_name=default
190 ... shard_type=config
191 ClusterManagement.Stop Single Member 3 msg=up: ODL1, ODL2, ODL3, down=none
192 BuiltIn.Run Keyword And Ignore Error
193 ... ClusterManagement.Get Raft State Of Shard Of All Member Nodes
194 ... shard_name=default
195 ... shard_type=config
197 Create Router router_2
198 [Documentation] Create Router and Add Interface to the subnets.
199 OpenStackOperations.Create Router ${ROUTERS}[1]
200 [Teardown] Report_Failure_Due_To_Bug 6117
202 Create Router router_3
203 [Documentation] Create Router and Add Interface to the subnets.
204 OpenStackOperations.Create Router ${ROUTERS}[2]
206 Add Interfaces To Router
207 [Documentation] Add Interfaces
208 FOR ${interface} IN @{SUBNETS}
209 OpenStackOperations.Add Router Interface ${ROUTERS}[2] ${interface}
212 Verify Created Routers
213 [Documentation] Check created routers using northbound rest calls
214 ${data} = Utils.Get Data From URI 1 ${NEUTRON_ROUTERS_API}
216 Should Contain ${data} ${ROUTERS}[2]
219 [Documentation] Bring up ODL3 again
220 BuiltIn.Run Keyword And Ignore Error
221 ... ClusterManagement.Get Raft State Of Shard Of All Member Nodes
222 ... shard_name=default
223 ... shard_type=config
224 ClusterManagement.Start Single Member 3 msg=up: ODL1, ODL2, down: ODL3
225 BuiltIn.Run Keyword And Ignore Error
226 ... ClusterManagement.Get Raft State Of Shard Of All Member Nodes
227 ... shard_name=default
228 ... shard_type=config
230 Ping Vm Instance1 In net_2 From net_1
231 [Documentation] Check reachability of vm instances by pinging to them after creating routers.
232 OpenStackOperations.Ping Vm From DHCP Namespace ${NETWORKS}[0] ${NET_2_L3_VM_IPS}[0]
234 Ping Vm Instance2 In net_2 From net_1
235 [Documentation] Check reachability of vm instances by pinging to them after creating routers.
236 OpenStackOperations.Ping Vm From DHCP Namespace ${NETWORKS}[0] ${NET_2_L3_VM_IPS}[1]
238 Ping Vm Instance3 In net_2 From net_1
239 [Documentation] Check reachability of vm instances by pinging to them after creating routers.
240 OpenStackOperations.Ping Vm From DHCP Namespace ${NETWORKS}[0] ${NET_2_L3_VM_IPS}[2]
242 Ping Vm Instance1 In net_1 From net_2
243 [Documentation] Check reachability of vm instances by pinging to them after creating routers.
244 OpenStackOperations.Ping Vm From DHCP Namespace ${NETWORKS}[1] ${NET_1_L3_VM_IPS}[0]
246 Ping Vm Instance2 In net_1 From net_2
247 [Documentation] Check reachability of vm instances by pinging to them after creating routers.
248 OpenStackOperations.Ping Vm From DHCP Namespace ${NETWORKS}[1] ${NET_1_L3_VM_IPS}[1]
250 Ping Vm Instance3 In net_1 From net_2
251 [Documentation] Check reachability of vm instances by pinging to them after creating routers.
252 OpenStackOperations.Ping Vm From DHCP Namespace ${NETWORKS}[1] ${NET_1_L3_VM_IPS}[2]
254 Connectivity Tests From Vm Instance1 In net_1 In Healthy Cluster
255 [Documentation] ssh to the VM instance and test operations.
256 ${dst_list} = BuiltIn.Create List @{NET_2_L3_VM_IPS} @{NET_1_L3_VM_IPS}
257 OpenStackOperations.Test Operations From Vm Instance ${NETWORKS}[0] ${NET_1_L3_VM_IPS}[0] ${dst_list}
258 [Teardown] OpenStackOperations.Get Test Teardown Debugs fail=False
260 Connectivity Tests From Vm Instance2 In net_1 In Healthy Cluster
261 [Documentation] ssh to the VM instance and test operations.
262 ${dst_list} = BuiltIn.Create List @{NET_2_L3_VM_IPS} @{NET_1_L3_VM_IPS}
263 OpenStackOperations.Test Operations From Vm Instance ${NETWORKS}[0] ${NET_1_L3_VM_IPS}[1] ${dst_list}
264 [Teardown] OpenStackOperations.Get Test Teardown Debugs fail=False
266 Connectivity Tests From Vm Instance3 In net_1 In Healthy Cluster
267 [Documentation] ssh to the VM instance and test operations.
268 ${dst_list} = BuiltIn.Create List @{NET_2_L3_VM_IPS} @{NET_1_L3_VM_IPS}
269 OpenStackOperations.Test Operations From Vm Instance ${NETWORKS}[0] ${NET_1_L3_VM_IPS}[2] ${dst_list}
270 [Teardown] OpenStackOperations.Get Test Teardown Debugs fail=False
272 Take Down ODL1 and ODL2
273 [Documentation] Stop the karaf in First and Second Controller
274 BuiltIn.Run Keyword And Ignore Error
275 ... ClusterManagement.Get Raft State Of Shard Of All Member Nodes
276 ... shard_name=default
277 ... shard_type=config
278 ClusterManagement.Stop Single Member 1 msg=up: ODL1, ODL2, ODL3, down=none
279 ClusterManagement.Stop Single Member 2 msg=up: ODL2, ODL3, down=ODL1
280 BuiltIn.Run Keyword And Ignore Error
281 ... ClusterManagement.Get Raft State Of Shard Of All Member Nodes
282 ... shard_name=default
283 ... shard_type=config
284 [Teardown] OpenStackOperations.Get Test Teardown Debugs fail=False
286 Connectivity Tests From Vm Instance1 In net_1 With Two ODLs Down
287 [Documentation] ssh to the VM instance and test operations.
288 ${dst_list} = BuiltIn.Create List @{NET_2_L3_VM_IPS} @{NET_1_L3_VM_IPS}
289 OpenStackOperations.Test Operations From Vm Instance ${NETWORKS}[0] ${NET_1_L3_VM_IPS}[0] ${dst_list}
290 [Teardown] OpenStackOperations.Get Test Teardown Debugs fail=False
292 Connectivity Tests From Vm Instance2 In net_1 With Two ODLs Down
293 [Documentation] ssh to the VM instance and test operations.
294 ${dst_list} = BuiltIn.Create List @{NET_2_L3_VM_IPS} @{NET_1_L3_VM_IPS}
295 OpenStackOperations.Test Operations From Vm Instance ${NETWORKS}[0] ${NET_1_L3_VM_IPS}[1] ${dst_list}
296 [Teardown] OpenStackOperations.Get Test Teardown Debugs fail=False
298 Connectivity Tests From Vm Instance3 In net_1 With Two ODLs Down
299 [Documentation] ssh to the VM instance and test operations.
300 ${dst_list} = BuiltIn.Create List @{NET_2_L3_VM_IPS} @{NET_1_L3_VM_IPS}
301 OpenStackOperations.Test Operations From Vm Instance ${NETWORKS}[0] ${NET_1_L3_VM_IPS}[2] ${dst_list}
302 [Teardown] OpenStackOperations.Get Test Teardown Debugs fail=False
304 Bring Up ODL1 and ODL2
305 [Documentation] Bring up ODL1 and ODL2 again. Do not check for cluster sync until all nodes are
306 ... up. akka will not let nodes join until they are all back up if two were down.
307 BuiltIn.Run Keyword And Ignore Error
308 ... ClusterManagement.Get Raft State Of Shard Of All Member Nodes
309 ... shard_name=default
310 ... shard_type=config
311 ClusterManagement.Start Single Member 1 msg=up: ODL3, down: ODL1, ODL2 wait_for_sync=False
312 BuiltIn.Run Keyword And Ignore Error
313 ... ClusterManagement.Get Raft State Of Shard Of All Member Nodes
314 ... shard_name=default
315 ... shard_type=config
316 ClusterManagement.Start Single Member 2 msg=up: ODL1, ODL3, down: ODL2
317 BuiltIn.Run Keyword And Ignore Error
318 ... ClusterManagement.Get Raft State Of Shard Of All Member Nodes
319 ... shard_name=default
320 ... shard_type=config
321 [Teardown] OpenStackOperations.Get Test Teardown Debugs fail=False
323 Take Down ODL2 and ODL3
324 [Documentation] Stop the karaf in First and Second Controller
325 BuiltIn.Run Keyword And Ignore Error
326 ... ClusterManagement.Get Raft State Of Shard Of All Member Nodes
327 ... shard_name=default
328 ... shard_type=config
329 ClusterManagement.Stop Single Member 2 msg=up: ODL1, ODL2, ODL3, down=none
330 BuiltIn.Run Keyword And Ignore Error
331 ... ClusterManagement.Get Raft State Of Shard Of All Member Nodes
332 ... shard_name=default
333 ... shard_type=config
334 ClusterManagement.Stop Single Member 3 msg=up: ODL1, ODL3, down=ODL2
335 BuiltIn.Run Keyword And Ignore Error
336 ... ClusterManagement.Get Raft State Of Shard Of All Member Nodes
337 ... shard_name=default
338 ... shard_type=config
339 [Teardown] OpenStackOperations.Get Test Teardown Debugs fail=False
341 Connectivity Tests From Vm Instance1 In net_2
342 [Documentation] ssh to the VM instance and test operations.
343 ${dst_list} = BuiltIn.Create List @{NET_2_L3_VM_IPS} @{NET_1_L3_VM_IPS}
344 BuiltIn.Wait Until Keyword Succeeds
347 ... OpenStackOperations.Test Operations From Vm Instance
349 ... ${NET_2_L3_VM_IPS}[0]
351 [Teardown] OpenStackOperations.Get Test Teardown Debugs fail=False
353 Connectivity Tests From Vm Instance2 In net_2
354 [Documentation] ssh to the VM instance and test operations.
355 ${dst_list} = BuiltIn.Create List @{NET_2_L3_VM_IPS} @{NET_1_L3_VM_IPS}
356 BuiltIn.Wait Until Keyword Succeeds
359 ... OpenStackOperations.Test Operations From Vm Instance
361 ... ${NET_2_L3_VM_IPS}[1]
363 [Teardown] OpenStackOperations.Get Test Teardown Debugs fail=False
365 Connectivity Tests From Vm Instance3 In net_2
366 [Documentation] ssh to the VM instance and test operations.
367 ${dst_list} = BuiltIn.Create List @{NET_2_L3_VM_IPS} @{NET_1_L3_VM_IPS}
368 BuiltIn.Wait Until Keyword Succeeds
371 ... OpenStackOperations.Test Operations From Vm Instance
373 ... ${NET_2_L3_VM_IPS}[2]
375 [Teardown] OpenStackOperations.Get Test Teardown Debugs fail=False
377 Bring Up ODL2 and ODL3
378 [Documentation] Bring up ODL2 and ODL3 again. Do not check for cluster sync until all nodes are
379 ... up. akka will not let nodes join until they are all back up if two were down.
380 BuiltIn.Run Keyword And Ignore Error
381 ... ClusterManagement.Get Raft State Of Shard Of All Member Nodes
382 ... shard_name=default
383 ... shard_type=config
384 ClusterManagement.Start Single Member 2 msg=up: ODL1, down: ODL2, ODL3 wait_for_sync=False
385 BuiltIn.Run Keyword And Ignore Error
386 ... ClusterManagement.Get Raft State Of Shard Of All Member Nodes
387 ... shard_name=default
388 ... shard_type=config
389 ClusterManagement.Start Single Member 3 msg=up: ODL1, ODL2, down: ODL3
390 BuiltIn.Run Keyword And Ignore Error
391 ... ClusterManagement.Get Raft State Of Shard Of All Member Nodes
392 ... shard_name=default
393 ... shard_type=config
394 [Teardown] OpenStackOperations.Get Test Teardown Debugs fail=False
396 Take Down All Instances
397 [Documentation] Stop karaf on all controllers
398 BuiltIn.Run Keyword And Ignore Error
399 ... ClusterManagement.Get Raft State Of Shard Of All Member Nodes
400 ... shard_name=default
401 ... shard_type=config
402 ClusterManagement.Stop_Members_From_List_Or_All
403 BuiltIn.Run Keyword And Ignore Error
404 ... ClusterManagement.Get Raft State Of Shard Of All Member Nodes
405 ... shard_name=default
406 ... shard_type=config
407 [Teardown] OpenStackOperations.Get Test Teardown Debugs fail=False
409 Bring Up All Instances
410 [Documentation] Bring up all controllers. Do not check for cluster sync until all nodes are
411 ... up. akka will not let nodes join until they are all back up if two were down.
412 BuiltIn.Run Keyword And Ignore Error
413 ... ClusterManagement.Get Raft State Of Shard Of All Member Nodes
414 ... shard_name=default
415 ... shard_type=config
416 ClusterManagement.Start Members From List Or All
417 BuiltIn.Run Keyword And Ignore Error
418 ... ClusterManagement.Get Raft State Of Shard Of All Member Nodes
419 ... shard_name=default
420 ... shard_type=config
421 [Teardown] OpenStackOperations.Get Test Teardown Debugs fail=False
423 Connectivity Tests From Vm Instance2 In net_2 after recovering all nodes
424 [Documentation] ssh to the VM instance and test operations.
425 ${dst_list} = BuiltIn.Create List @{NET_2_L3_VM_IPS} @{NET_1_L3_VM_IPS}
426 OpenStackOperations.Test Operations From Vm Instance ${NETWORKS}[1] ${NET_2_L3_VM_IPS}[1] ${dst_list}
427 [Teardown] OpenStackOperations.Get Test Teardown Debugs fail=False
429 Connectivity Tests From Vm Instance3 In net_2 after recovering all nodes
430 [Documentation] ssh to the VM instance and test operations.
431 ${dst_list} = BuiltIn.Create List @{NET_2_L3_VM_IPS} @{NET_1_L3_VM_IPS}
432 OpenStackOperations.Test Operations From Vm Instance ${NETWORKS}[1] ${NET_2_L3_VM_IPS}[2] ${dst_list}
433 [Teardown] OpenStackOperations.Get Test Teardown Debugs fail=False
435 Delete Vm Instances In net_1
436 [Documentation] Delete Vm instances using instance names in net_1.
437 FOR ${vm} IN @{NET_1_VMS}
438 OpenStackOperations.Delete Vm Instance ${vm}
441 Delete Vm Instances In net_2
442 [Documentation] Delete Vm instances using instance names in net_2.
443 FOR ${vm} IN @{NET_2_VMS}
444 OpenStackOperations.Delete Vm Instance ${vm}
447 Delete Router Interfaces
448 [Documentation] Remove Interface to the subnets.
449 FOR ${interface} IN @{SUBNETS}
450 OpenStackOperations.Remove Interface ${ROUTERS}[2] ${interface}
454 [Documentation] Delete Router and Interface to the subnets.
455 OpenStackOperations.Delete Router ${ROUTERS}[1]
456 OpenStackOperations.Delete Router ${ROUTERS}[2]
458 Verify Deleted Routers
459 [Documentation] Check deleted routers using northbound rest calls
460 ${data} = Utils.Get Data From URI 1 ${NEUTRON_ROUTERS_API}
462 BuiltIn.Should Not Contain ${data} ${ROUTERS}[2]
464 Delete Sub Network In net_1
465 [Documentation] Delete Sub Nets for the Networks with neutron request.
466 OpenStackOperations.Delete SubNet ${SUBNETS}[0]
468 Delete Sub Network In net_2
469 [Documentation] Delete Sub Nets for the Networks with neutron request.
470 OpenStackOperations.Delete SubNet ${SUBNETS}[1]
473 [Documentation] Delete Networks with neutron request.
474 FOR ${network} IN @{NETWORKS}
475 OpenStackOperations.Delete Network ${network}
478 Delete Security Group
479 [Documentation] Delete security groups with neutron request
480 OpenStackOperations.Delete SecurityGroup ${SECURITY_GROUP}
483 [Documentation] Verify that flows have been cleaned up properly after removing all neutron configurations
484 DataModels.Verify Flows Are Cleaned Up On All OpenStack Nodes