3 Library RequestsLibrary
5 Library ClusterStateLibrary.py
8 ${smc_node} /org.opendaylight.controller:Category=ShardManager,name=shard-manager-config,type=DistributedConfigDatastore
12 [Arguments] ${exclude_controller}=${EMPTY}
13 [Documentation] Creates a list of all controllers minus any excluded controller.
14 Log ${exclude_controller}
15 @{searchlist} Create List ${CONTROLLER} ${CONTROLLER1} ${CONTROLLER2}
16 Remove Values From List ${searchlist} ${exclude_controller}
18 [Return] ${searchlist}
21 [Arguments] ${shard_name} ${old_leader}=${EMPTY}
22 [Documentation] Returns the IP addr or hostname of the leader of the specified shard.
23 ... Controllers are specifed in the pybot command line.
24 ${searchlist} Get Controller List ${old_leader}
25 ${leader} GetLeader ${shard_name} ${3} ${3} ${1} ${PORT}
27 Should Not Be Equal As Strings ${leader} None
28 Run Keyword If '${old_leader}'!='${EMPTY}' Should Not Be Equal ${old_leader} ${leader}
31 Wait For Leader To Be Found
32 [Arguments] ${shard_name}
33 [Documentation] Waits until the leader of the specified shard is found.
34 ${leader} Wait Until Keyword Succeeds 12s 2s Get Leader And Verify ${shard_name}
39 [Arguments] ${shard_name} ${current_leader}
40 [Documentation] Forces a change of leadership by shutting down the current leader.
41 Stop One Or More Controllers ${current_leader}
42 ${new_leader} Wait Until Keyword Succeeds 60s 2s Get Leader And Verify ${shard_name} ${current_leader}
43 # TODO: Future enhanement: make sure the other controller is a follower and not a master or candidate.
45 [Return] ${new_leader}
48 [Arguments] ${shard_name} ${exclude_controller}=${EMPTY}
49 [Documentation] Returns the IP addresses or hostnames of all followers of the specified shard.
50 ${searchlist} Get Controller List ${exclude_controller}
51 ${followers} GetFollowers ${shard_name} ${3} ${3} ${1} ${PORT}
54 Should Not Be Empty ${followers}
58 [Arguments] ${controller_ip} ${num_cars} ${timeout}=12s
59 [Documentation] Initializes shard and then adds the specified number of cars and performs a GET as a check.
60 ${resp} InitCar ${controller_ip} ${PORT}
61 Should Be Equal As Strings ${resp.status_code} 204
62 ${resp} AddCar ${controller_ip} ${RESTCONFPORT} ${num_cars} 204
63 Should Be Equal As Strings ${resp.status_code} 204
64 Wait Until Keyword Succeeds ${timeout} 2s Get Cars And Verify ${controller_ip} ${num_cars}
66 Add Cars And Verify Without Init
67 [Arguments] ${controller_ip} ${num_cars} ${timeout}=12s
68 [Documentation] Adds cars to an initialized cars shard then performs a GET as a check.
69 Comment First car add may return 409, but subsequent should be 204
70 ${resp} AddCar ${controller_ip} ${RESTCONFPORT} ${num_cars} 204 409
71 Should Be Equal As Strings ${resp.status_code} 204
72 Wait Until Keyword Succeeds ${timeout} 2s Get Cars And Verify ${controller_ip} ${num_cars}
75 [Arguments] ${controller_ip} ${num_cars}
76 [Documentation] Gets cars and verifies that the manufacturer is correct.
77 # TODO: Future enhanement: verify all fields.
78 ${resp} Getcars ${controller_ip} ${RESTCONFPORT} ${0}
79 Should Be Equal As Strings ${resp.status_code} 200
80 : FOR ${i} IN RANGE 1 ${num_cars}+1
81 \ Should Contain ${resp.content} manufacturer${i}
84 [Arguments] ${controller_ip} ${num_people}
85 [Documentation] Note: The first AddPerson call passed with 0 posts directly to the data store to get
86 ... the people container created so the subsequent AddPerson RPC calls that put to the
87 ... person list will succeed.
88 ${resp} AddPerson ${controller_ip} ${RESTCONFPORT} ${0} 204
89 Should Be Equal As Strings ${resp.status_code} 204
90 Wait Until Keyword Succeeds 12s 2s Get One Person And Verify ${controller_ip} ${0}
91 ${resp} AddPerson ${controller_ip} ${RESTCONFPORT} ${num_people} 200
92 Wait Until Keyword Succeeds 12s 2s Get People And Verify ${controller_ip} ${num_people}
94 Get One Person And Verify
95 [Arguments] ${controller_ip} ${number}
96 [Documentation] Gets a person and verifies that the user ID is correct.
97 # TODO: Future enhanement: verify all fields.
98 ${resp} GetPersons ${controller_ip} ${RESTCONFPORT} ${0}
99 Should Be Equal As Strings ${resp.status_code} 200
100 Should Contain ${resp.content} user${number}
102 Get People And Verify
103 [Arguments] ${controller_ip} ${num_people}
104 [Documentation] Gets multiple people and verifies that the user IDs are correct.
105 # TODO: Future enhanement: verify all fields.
106 ${resp} GetPersons ${controller_ip} ${RESTCONFPORT} ${0}
107 Should Be Equal As Strings ${resp.status_code} 200
108 : FOR ${i} IN RANGE 1 ${num_people}+1
109 \ Should Contain ${resp.content} user${i}
111 Add Car Person And Verify
112 [Arguments] ${controller_ip}
113 [Documentation] Add a car-person via the data store and get the car-person from Leader.
114 ... Note: This is done to get the car-people container created so subsequent
115 ... BuyCar RPC puts to the car-person list will succeed.
116 AddCarPerson ${controller_ip} ${RESTCONFPORT} ${0}
117 Wait Until Keyword Succeeds 60s 2s Get One Car-Person Mapping And Verify ${controller_ip} ${0}
119 Get One Car-Person Mapping And Verify
120 [Arguments] ${controller_ip} ${number}
121 [Documentation] Gets a car person mapping and verifies that the user ID is correct.
122 ${resp} GetCarPersonMappings ${controller_ip} ${RESTCONFPORT} ${0}
123 Should Be Equal As Strings ${resp.status_code} 200
124 Should Contain ${resp.content} user${number}
126 Get Car-Person Mappings And Verify
127 [Arguments] ${controller_ip} ${num_entries}
128 ${resp} GetCarPersonMappings ${controller_ip} ${RESTCONFPORT} ${0}
129 Should Be Equal As Strings ${resp.status_code} 200
130 : FOR ${i} IN RANGE 1 ${num_entries}+1
131 \ Should Contain ${resp.content} user${i}
134 [Arguments] ${controller_ip} ${num_entries} ${start}=${0}
135 Wait Until Keyword Succeeds 60s 2s BuyCar ${controller_ip} ${RESTCONFPORT} ${num_entries}
139 [Arguments] ${controller_ip}
140 ${resp} Getcars ${controller_ip} ${RESTCONFPORT} ${0}
141 Should Be Equal As Strings ${resp.status_code} 404
143 Delete All Cars And Verify
144 [Arguments] ${controller_ip}
145 DeleteAllCars ${controller_ip} ${RESTCONFPORT} ${0}
146 Wait Until Keyword Succeeds 60s 2s Check Cars Deleted ${controller_ip}
149 [Arguments] ${controller_ip}
150 ${resp} GetPersons ${controller_ip} ${RESTCONFPORT} ${0}
151 Should Be Equal As Strings ${resp.status_code} 404
153 Delete All People And Verify
154 [Arguments] ${controller_ip}
155 DeleteAllPersons ${controller_ip} ${RESTCONFPORT} ${0}
156 Wait Until Keyword Succeeds 60s 2s Check People Deleted ${controller_ip}
158 Check Cars-Persons Deleted
159 [Arguments] ${controller_ip}
160 ${resp} GetCarPersonMappings ${controller_ip} ${RESTCONFPORT} ${0}
161 Should Be Equal As Strings ${resp.status_code} 404
163 Delete All Cars-Persons And Verify
164 [Arguments] ${controller_ip}
165 DeleteAllCarsPersons ${controller_ip} ${RESTCONFPORT} ${0}
166 Wait Until Keyword Succeeds 60s 2s Check Cars-Persons Deleted ${controller_ip}
168 Stop One Or More Controllers
169 [Arguments] @{controllers}
170 [Documentation] Give this keyword a scalar or list of controllers to be stopped.
171 ${cmd} = Set Variable ${KARAF_HOME}/bin/stop
172 : FOR ${ip} IN @{controllers}
173 \ Run Command On Remote System ${ip} ${cmd}
174 : FOR ${ip} IN @{controllers}
175 \ Wait Until Keyword Succeeds ${CONTROLLER_STOP_TIMEOUT} s 3 s Controller Down Check ${ip}
177 Start One Or More Controllers
178 [Arguments] @{controllers}
179 [Documentation] Give this keyword a scalar or list of controllers to be started.
180 ${cmd} = Set Variable ${KARAF_HOME}/bin/start
181 : FOR ${ip} IN @{controllers}
182 \ Run Command On Remote System ${ip} ${cmd}
183 # TODO: This should throw an error if controller never comes up.
184 : FOR ${ip} IN @{controllers}
185 \ UtilLibrary.Wait For Controller Up ${ip} ${RESTCONFPORT}
187 Kill One Or More Controllers
188 [Arguments] @{controllers}
189 [Documentation] Give this keyword a scalar or list of controllers to be stopped.
190 ${cmd} = Set Variable ps axf | grep karaf | grep -v grep | awk '{print \"kill -9 \" $1}' | sh
192 : FOR ${ip} IN @{controllers}
193 \ Run Command On Remote System ${ip} ${cmd}
194 : FOR ${ip} IN @{controllers}
195 \ Wait Until Keyword Succeeds 12 s 3 s Controller Down Check ${ip}
197 Controller Down Check
199 [Documentation] Checks to see if a controller is down by verifying that the karaf process isn't present.
200 ${cmd} = Set Variable ps axf | grep karaf | grep -v grep | wc -l
201 ${response} Run Command On Remote System ${ip} ${cmd}
202 Log Number of controller instances running: ${response}
203 Should Start With ${response} 0 Controller process found or there may be extra instances of karaf running on the host machine.
205 Clean One Or More Journals
206 [Arguments] @{controllers}
207 [Documentation] Give this keyword a scalar or list of controllers on which to clean journals.
208 ${del_cmd} = Set Variable rm -rf ${KARAF_HOME}/journal
209 : FOR ${ip} IN @{controllers}
210 \ Run Command On Remote System ${ip} ${del_cmd}
212 Clean One Or More Snapshots
213 [Arguments] @{controllers}
214 [Documentation] Give this keyword a scalar or list of controllers on which to clean snapshots.
215 ${del_cmd} = Set Variable rm -rf ${KARAF_HOME}/snapshots
216 : FOR ${ip} IN @{controllers}
217 \ Run Command On Remote System ${ip} ${del_cmd}
219 Show Cluster Configuation Files
220 [Arguments] @{controllers}
221 [Documentation] Prints out the cluster configuration files for one or more controllers.
222 Log controllers: @{controllers}
223 ${cmd} = Set Variable cat ${KARAF_HOME}/configuration/initial/akka.conf
224 : FOR ${ip} IN @{controllers}
225 \ Run Command On Remote System ${ip} ${cmd}
226 ${cmd} = Set Variable cat ${KARAF_HOME}/configuration/initial/modules.conf
227 : FOR ${ip} IN @{controllers}
228 \ Run Command On Remote System ${ip} ${cmd}
229 ${cmd} = Set Variable cat ${KARAF_HOME}/configuration/initial/module-shards.conf
230 : FOR ${ip} IN @{controllers}
231 \ Run Command On Remote System ${ip} ${cmd}
232 ${cmd} = Set Variable cat ${KARAF_HOME}/configuration/initial/jolokia.xml
233 : FOR ${ip} IN @{controllers}
234 \ Run Command On Remote System ${ip} ${cmd}
235 ${cmd} = Set Variable cat ${KARAF_HOME}/etc/initial/org.apache.karaf.management.cfg
236 : FOR ${ip} IN @{controllers}
237 \ Run Command On Remote System ${ip} ${cmd}
238 ${cmd} = Set Variable cat ${KARAF_HOME}/etc/org.apache.karaf.features.cfg
239 : FOR ${ip} IN @{controllers}
240 \ Run Command On Remote System ${ip} ${cmd}
243 [Arguments] ${selected controller} ${PORT} ${nth car}
244 [Documentation] Verifies that the first through nth car is present.
245 ${resp} Getcars ${selected controller} ${PORT} ${0}
246 Should Be Equal As Strings ${resp.status_code} 200
247 : FOR ${INDEX} IN RANGE 1 ${nth car}
248 \ ${counter}= Convert to String ${INDEX}
249 \ Log manufacturer${counter}
250 \ Should Contain ${resp.content} manufacturer${counter}
253 [Arguments] ${selected controller} ${PORT} ${nth person}
254 [Documentation] Verifies that the first through nth person is present.
255 ${resp} GetPersons ${selected controller} ${PORT} ${0}
256 Should Be Equal As Strings ${resp.status_code} 200
257 : FOR ${INDEX} IN RANGE 1 ${nth person}
258 \ ${counter}= Convert to String ${INDEX}
260 \ Should Contain ${resp.content} user${counter}
263 [Arguments] ${selected controller} ${PORT} ${nth carperson}
264 [Documentation] Verifies that the first through nth car-person is present.
265 ${resp} GetCarPersonMappings ${selected controller} ${PORT} ${0}
266 Should Be Equal As Strings ${resp.status_code} 200
267 : FOR ${INDEX} IN RANGE 1 ${nth carperson}
268 \ ${counter}= Convert to String ${INDEX}
270 \ Should Contain ${resp.content} user${counter}
272 Isolate a Controller From Cluster
273 [Arguments] ${isolated controller} @{controllers}
274 [Documentation] Use IPTables to isolate one controller from the cluster.
275 ... On the isolated controller it blocks IP traffic to and from each of the other controllers.
276 : FOR ${controller} IN @{controllers}
277 \ ${other controller}= Evaluate "${isolated controller}" != "${controller}"
278 \ Run Keyword If ${other controller} Isolate One Controller From Another ${isolated controller} ${controller}
280 Rejoin a Controller To Cluster
281 [Arguments] ${isolated controller} @{controllers}
282 [Documentation] Use IPTables to rejoin one controller to the cluster.
283 ... On the isolated controller it unblocks IP traffic to and from each of the other controllers.
284 : FOR ${controller} IN @{controllers}
285 \ ${other controller}= Evaluate "${isolated controller}" != "${controller}"
286 \ Run Keyword If ${other controller} Rejoin One Controller To Another ${isolated controller} ${controller}
288 Isolate One Controller From Another
289 [Arguments] ${isolated controller} ${controller}
290 [Documentation] Inserts an IPTable rule to disconnect one controller from another controller in the cluster.
291 Modify IPTables ${isolated controller} ${controller} -I
293 Rejoin One Controller To Another
294 [Arguments] ${isolated controller} ${controller}
295 [Documentation] Deletes an IPTable rule, allowing one controller to reconnect to another controller in the cluster.
296 Modify IPTables ${isolated controller} ${controller} -D
299 [Arguments] ${isolated controller} ${controller} ${rule type}
300 [Documentation] Adds a rule, usually inserting or deleting an entry between two controllers.
301 ${base string} Set Variable sudo iptables ${rule type} OUTPUT -p all --source
302 ${cmd string} Catenate ${base string} ${isolated controller} --destination ${controller} -j DROP
303 Run Command On Remote System ${isolated controller} ${cmd string}
304 ${cmd string} Catenate ${base string} ${controller} --destination ${isolated controller} -j DROP
305 Run Command On Remote System ${isolated controller} ${cmd string}
306 ${cmd string} Set Variable sudo iptables -L -n
307 ${return string}= Run Command On Remote System ${isolated controller} ${cmd string}
309 Run Keyword If "${rule type}" == '-I' Should Match Regexp ${return string} [\s\S]*DROP *all *-- *${isolated controller} *${controller}[\s\S]*
310 Run Keyword If "${rule type}" == '-I' Should Match Regexp ${return string} [\s\S]*DROP *all *-- *${controller} *${isolated controller}[\s\S]*
312 Run Keyword If "${rule type}" == '-D' Should Match Regexp ${return string} (?![\s\S]*DROP *all *-- *${isolated controller} *${controller}[\s\S]*)
313 Run Keyword If "${rule type}" == '-D' Should Match Regexp ${return string} (?![\s\S]*DROP *all *-- *${controller} *${isolated controller}[\s\S]*)
315 Rejoin All Isolated Controllers
316 [Arguments] @{controllers}
317 [Documentation] Wipe all IPTables rules from all controllers, thus rejoining all controllers.
318 : FOR ${isolated controller} IN @{controllers}
319 \ Flush IPTables ${isolated controller}
322 [Arguments] ${isolated controller}
323 [Documentation] This keyword is generally not called from a test case but supports a complete wipe of all rules on
325 ${cmd string} Set Variable sudo iptables -v -F
326 ${return string}= Run Command On Remote System ${isolated controller} ${cmd string}
327 Log return: ${return string}
328 Should Contain ${return string} Flushing chain `INPUT'
329 Should Contain ${return string} Flushing chain `FORWARD'
330 Should Contain ${return string} Flushing chain `OUTPUT'
332 Wait for Cluster Sync
333 [Arguments] ${timeout} @{controllers}
334 [Documentation] Waits for one or more clustered controllers to report Sync Status as true.
335 : FOR ${ip} IN @{controllers}
336 \ ${resp}= Wait Until Keyword Succeeds ${timeout} 2s Controller Sync Status Should Be True ${ip}
338 Controller Sync Status Should Be True
340 [Documentation] Checks if Sync Status is true.
341 ${SyncStatus}= Get Controller Sync Status ${ip}
342 Should Be Equal ${SyncStatus} ${True}
344 Controller Sync Status Should Be False
346 [Documentation] Checks if Sync Status is false.
347 ${SyncStatus}= Get Controller Sync Status ${ip}
348 Should Be Equal ${SyncStatus} ${False}
350 Get Controller Sync Status
351 [Arguments] ${controller_ip}
352 [Documentation] Return Sync Status.
353 ${api} Set Variable /jolokia/read
354 Create_Session session http://${controller_ip}:${RESTCONFPORT}${api} headers=${HEADERS} auth=${AUTH}
355 ${resp}= RequestsLibrary.Get session ${smc_node}
358 ${json}= Set Variable ${resp.json()}
359 ${value}= Get From Dictionary ${json} value
361 ${SyncStatus}= Get From Dictionary ${value} SyncStatus
362 Log SyncSatus: ${SyncStatus}
363 [Return] ${SyncStatus}