4 Library ClusterStateLibrary.py
10 [Arguments] ${exclude_controller}=${EMPTY}
11 [Documentation] Creates a list of all controllers minus any excluded controller.
12 Log ${exclude_controller}
13 @{searchlist} Create List ${CONTROLLER} ${CONTROLLER1} ${CONTROLLER2}
14 Remove Values From List ${searchlist} ${exclude_controller}
16 [Return] ${searchlist}
19 [Arguments] ${shard_name} ${old_leader}=${EMPTY}
20 [Documentation] Returns the IP addr or hostname of the leader of the specified shard.
21 ... Controllers are specifed in the pybot command line.
22 ${searchlist} Get Controller List ${old_leader}
23 ${leader} GetLeader ${shard_name} ${3} ${3} ${1} ${PORT}
25 Should Not Be Equal As Strings ${leader} None
26 Run Keyword If '${old_leader}'!='${EMPTY}' Should Not Be Equal ${old_leader} ${leader}
29 Wait For Leader To Be Found
30 [Arguments] ${shard_name}
31 [Documentation] Waits until the leader of the specified shard is found.
32 ${leader} Wait Until Keyword Succeeds 12s 2s Get Leader And Verify ${shard_name}
37 [Arguments] ${shard_name} ${current_leader}
38 [Documentation] Forces a change of leadership by shutting down the current leader.
39 Stop One Or More Controllers ${current_leader}
40 ${new_leader} Wait Until Keyword Succeeds 60s 2s Get Leader And Verify ${shard_name} ${current_leader}
41 # TODO: Future enhanement: make sure the other controller is a follower and not a master or candidate.
43 [Return] ${new_leader}
46 [Arguments] ${shard_name} ${exclude_controller}=${EMPTY}
47 [Documentation] Returns the IP addresses or hostnames of all followers of the specified shard.
48 ${searchlist} Get Controller List ${exclude_controller}
49 ${followers} GetFollowers ${shard_name} ${3} ${3} ${1} ${PORT}
52 Should Not Be Empty ${followers}
56 [Arguments] ${controller_ip} ${num_cars} ${timeout}=12s
57 [Documentation] Initializes shard and then adds the specified number of cars and performs a GET as a check.
58 ${resp} InitCar ${controller_ip} ${PORT}
59 Should Be Equal As Strings ${resp.status_code} 204
60 ${resp} AddCar ${controller_ip} ${RESTCONFPORT} ${num_cars} 204
61 Should Be Equal As Strings ${resp.status_code} 204
62 Wait Until Keyword Succeeds ${timeout} 2s Get Cars And Verify ${controller_ip} ${num_cars}
64 Add Cars And Verify Without Init
65 [Arguments] ${controller_ip} ${num_cars} ${timeout}=12s
66 [Documentation] Adds cars to an initialized cars shard then performs a GET as a check.
67 Comment First car add may return 409, but subsequent should be 204
68 ${resp} AddCar ${controller_ip} ${RESTCONFPORT} ${num_cars} 204 409
69 Should Be Equal As Strings ${resp.status_code} 204
70 Wait Until Keyword Succeeds ${timeout} 2s Get Cars And Verify ${controller_ip} ${num_cars}
73 [Arguments] ${controller_ip} ${num_cars}
74 [Documentation] Gets cars and verifies that the manufacturer is correct.
75 # TODO: Future enhanement: verify all fields.
76 ${resp} Getcars ${controller_ip} ${RESTCONFPORT} ${0}
77 Should Be Equal As Strings ${resp.status_code} 200
78 : FOR ${i} IN RANGE 1 ${num_cars}+1
79 \ Should Contain ${resp.content} manufacturer${i}
82 [Arguments] ${controller_ip} ${num_people}
83 [Documentation] Note: The first AddPerson call passed with 0 posts directly to the data store to get
84 ... the people container created so the subsequent AddPerson RPC calls that put to the
85 ... person list will succeed.
86 ${resp} AddPerson ${controller_ip} ${RESTCONFPORT} ${0} 204
87 Should Be Equal As Strings ${resp.status_code} 204
88 Wait Until Keyword Succeeds 12s 2s Get One Person And Verify ${controller_ip} ${0}
89 ${resp} AddPerson ${controller_ip} ${RESTCONFPORT} ${num_people} 200
90 Wait Until Keyword Succeeds 12s 2s Get People And Verify ${controller_ip} ${num_people}
92 Get One Person And Verify
93 [Arguments] ${controller_ip} ${number}
94 [Documentation] Gets a person and verifies that the user ID is correct.
95 # TODO: Future enhanement: verify all fields.
96 ${resp} GetPersons ${controller_ip} ${RESTCONFPORT} ${0}
97 Should Be Equal As Strings ${resp.status_code} 200
98 Should Contain ${resp.content} user${number}
100 Get People And Verify
101 [Arguments] ${controller_ip} ${num_people}
102 [Documentation] Gets multiple people and verifies that the user IDs are correct.
103 # TODO: Future enhanement: verify all fields.
104 ${resp} GetPersons ${controller_ip} ${RESTCONFPORT} ${0}
105 Should Be Equal As Strings ${resp.status_code} 200
106 : FOR ${i} IN RANGE 1 ${num_people}+1
107 \ Should Contain ${resp.content} user${i}
109 Add Car Person And Verify
110 [Arguments] ${controller_ip}
111 [Documentation] Add a car-person via the data store and get the car-person from Leader.
112 ... Note: This is done to get the car-people container created so subsequent
113 ... BuyCar RPC puts to the car-person list will succeed.
114 AddCarPerson ${controller_ip} ${RESTCONFPORT} ${0}
115 Wait Until Keyword Succeeds 60s 2s Get One Car-Person Mapping And Verify ${controller_ip} ${0}
117 Get One Car-Person Mapping And Verify
118 [Arguments] ${controller_ip} ${number}
119 [Documentation] Gets a car person mapping and verifies that the user ID is correct.
120 ${resp} GetCarPersonMappings ${controller_ip} ${RESTCONFPORT} ${0}
121 Should Be Equal As Strings ${resp.status_code} 200
122 Should Contain ${resp.content} user${number}
124 Get Car-Person Mappings And Verify
125 [Arguments] ${controller_ip} ${num_entries}
126 ${resp} GetCarPersonMappings ${controller_ip} ${RESTCONFPORT} ${0}
127 Should Be Equal As Strings ${resp.status_code} 200
128 : FOR ${i} IN RANGE 1 ${num_entries}+1
129 \ Should Contain ${resp.content} user${i}
132 [Arguments] ${controller_ip} ${num_entries} ${start}=${0}
133 Wait Until Keyword Succeeds 60s 2s BuyCar ${controller_ip} ${RESTCONFPORT} ${num_entries}
137 [Arguments] ${controller_ip}
138 ${resp} Getcars ${controller_ip} ${RESTCONFPORT} ${0}
139 Should Be Equal As Strings ${resp.status_code} 404
141 Delete All Cars And Verify
142 [Arguments] ${controller_ip}
143 DeleteAllCars ${controller_ip} ${RESTCONFPORT} ${0}
144 Wait Until Keyword Succeeds 60s 2s Check Cars Deleted ${controller_ip}
147 [Arguments] ${controller_ip}
148 ${resp} GetPersons ${controller_ip} ${RESTCONFPORT} ${0}
149 Should Be Equal As Strings ${resp.status_code} 404
151 Delete All People And Verify
152 [Arguments] ${controller_ip}
153 DeleteAllPersons ${controller_ip} ${RESTCONFPORT} ${0}
154 Wait Until Keyword Succeeds 60s 2s Check People Deleted ${controller_ip}
156 Check Cars-Persons Deleted
157 [Arguments] ${controller_ip}
158 ${resp} GetCarPersonMappings ${controller_ip} ${RESTCONFPORT} ${0}
159 Should Be Equal As Strings ${resp.status_code} 404
161 Delete All Cars-Persons And Verify
162 [Arguments] ${controller_ip}
163 DeleteAllCarsPersons ${controller_ip} ${RESTCONFPORT} ${0}
164 Wait Until Keyword Succeeds 60s 2s Check Cars-Persons Deleted ${controller_ip}
166 Stop One Or More Controllers
167 [Arguments] @{controllers}
168 [Documentation] Give this keyword a scalar or list of controllers to be stopped.
169 ${cmd} = Set Variable ${KARAF_HOME}/bin/stop
170 : FOR ${ip} IN @{controllers}
171 \ Run Command On Remote System ${ip} ${cmd}
172 : FOR ${ip} IN @{controllers}
173 \ Wait Until Keyword Succeeds 120 s 3 s Controller Down Check ${ip}
175 Start One Or More Controllers
176 [Arguments] @{controllers}
177 [Documentation] Give this keyword a scalar or list of controllers to be started.
178 ${cmd} = Set Variable ${KARAF_HOME}/bin/start
179 : FOR ${ip} IN @{controllers}
180 \ Run Command On Remote System ${ip} ${cmd}
181 # TODO: This should throw an error if controller never comes up.
182 : FOR ${ip} IN @{controllers}
183 \ UtilLibrary.Wait For Controller Up ${ip} ${RESTCONFPORT}
185 Kill One Or More Controllers
186 [Arguments] @{controllers}
187 [Documentation] Give this keyword a scalar or list of controllers to be stopped.
188 ${cmd} = Set Variable ps axf | grep karaf | grep -v grep | awk '{print \"kill -9 \" $1}' | sh
190 : FOR ${ip} IN @{controllers}
191 \ Run Command On Remote System ${ip} ${cmd}
192 : FOR ${ip} IN @{controllers}
193 \ Wait Until Keyword Succeeds 12 s 3 s Controller Down Check ${ip}
195 Controller Down Check
197 [Documentation] Checks to see if a controller is down by verifying that the karaf process isn't present.
198 ${cmd} = Set Variable ps axf | grep karaf | grep -v grep | wc -l
199 ${response} Run Command On Remote System ${ip} ${cmd}
200 Log Number of controller instances running: ${response}
201 Should Start With ${response} 0 Controller process found or there may be extra instances of karaf running on the host machine.
203 Clean One Or More Journals
204 [Arguments] @{controllers}
205 [Documentation] Give this keyword a scalar or list of controllers on which to clean journals.
206 ${del_cmd} = Set Variable rm -rf ${KARAF_HOME}/journal
207 : FOR ${ip} IN @{controllers}
208 \ Run Command On Remote System ${ip} ${del_cmd}
210 Clean One Or More Snapshots
211 [Arguments] @{controllers}
212 [Documentation] Give this keyword a scalar or list of controllers on which to clean snapshots.
213 ${del_cmd} = Set Variable rm -rf ${KARAF_HOME}/snapshots
214 : FOR ${ip} IN @{controllers}
215 \ Run Command On Remote System ${ip} ${del_cmd}
217 Show Cluster Configuation Files
218 [Arguments] @{controllers}
219 [Documentation] Prints out the cluster configuration files for one or more controllers.
220 Log controllers: @{controllers}
221 ${cmd} = Set Variable cat ${KARAF_HOME}/configuration/initial/akka.conf
222 : FOR ${ip} IN @{controllers}
223 \ Run Command On Remote System ${ip} ${cmd}
224 ${cmd} = Set Variable cat ${KARAF_HOME}/configuration/initial/modules.conf
225 : FOR ${ip} IN @{controllers}
226 \ Run Command On Remote System ${ip} ${cmd}
227 ${cmd} = Set Variable cat ${KARAF_HOME}/configuration/initial/module-shards.conf
228 : FOR ${ip} IN @{controllers}
229 \ Run Command On Remote System ${ip} ${cmd}
230 ${cmd} = Set Variable cat ${KARAF_HOME}/configuration/initial/jolokia.xml
231 : FOR ${ip} IN @{controllers}
232 \ Run Command On Remote System ${ip} ${cmd}
233 ${cmd} = Set Variable cat ${KARAF_HOME}/etc/initial/org.apache.karaf.management.cfg
234 : FOR ${ip} IN @{controllers}
235 \ Run Command On Remote System ${ip} ${cmd}
236 ${cmd} = Set Variable cat ${KARAF_HOME}/etc/org.apache.karaf.features.cfg
237 : FOR ${ip} IN @{controllers}
238 \ Run Command On Remote System ${ip} ${cmd}
241 [Arguments] ${selected controller} ${PORT} ${nth car}
242 [Documentation] Verifies that the first through nth car is present.
243 ${resp} Getcars ${selected controller} ${PORT} ${0}
244 Should Be Equal As Strings ${resp.status_code} 200
245 : FOR ${INDEX} IN RANGE 1 ${nth car}
246 \ ${counter}= Convert to String ${INDEX}
247 \ Log manufacturer${counter}
248 \ Should Contain ${resp.content} manufacturer${counter}
251 [Arguments] ${selected controller} ${PORT} ${nth person}
252 [Documentation] Verifies that the first through nth person is present.
253 ${resp} GetPersons ${selected controller} ${PORT} ${0}
254 Should Be Equal As Strings ${resp.status_code} 200
255 : FOR ${INDEX} IN RANGE 1 ${nth person}
256 \ ${counter}= Convert to String ${INDEX}
258 \ Should Contain ${resp.content} user${counter}
261 [Arguments] ${selected controller} ${PORT} ${nth carperson}
262 [Documentation] Verifies that the first through nth car-person is present.
263 ${resp} GetCarPersonMappings ${selected controller} ${PORT} ${0}
264 Should Be Equal As Strings ${resp.status_code} 200
265 : FOR ${INDEX} IN RANGE 1 ${nth carperson}
266 \ ${counter}= Convert to String ${INDEX}
268 \ Should Contain ${resp.content} user${counter}
270 Isolate a Controller From Cluster
271 [Arguments] ${isolated controller} @{controllers}
272 [Documentation] Use IPTables to isolate one controller from the cluster.
273 ... On the isolated controller it blocks IP traffic to and from each of the other controllers.
274 : FOR ${controller} IN @{controllers}
275 \ ${other controller}= Evaluate "${isolated controller}" != "${controller}"
276 \ Run Keyword If ${other controller} Isolate One Controller From Another ${isolated controller} ${controller}
278 Rejoin a Controller To Cluster
279 [Arguments] ${isolated controller} @{controllers}
280 [Documentation] Use IPTables to rejoin one controller to the cluster.
281 ... On the isolated controller it unblocks IP traffic to and from each of the other controllers.
282 : FOR ${controller} IN @{controllers}
283 \ ${other controller}= Evaluate "${isolated controller}" != "${controller}"
284 \ Run Keyword If ${other controller} Rejoin One Controller To Another ${isolated controller} ${controller}
286 Isolate One Controller From Another
287 [Arguments] ${isolated controller} ${controller}
288 [Documentation] Inserts an IPTable rule to disconnect one controller from another controller in the cluster.
289 Modify IPTables ${isolated controller} ${controller} -I
291 Rejoin One Controller To Another
292 [Arguments] ${isolated controller} ${controller}
293 [Documentation] Deletes an IPTable rule, allowing one controller to reconnect to another controller in the cluster.
294 Modify IPTables ${isolated controller} ${controller} -D
297 [Arguments] ${isolated controller} ${controller} ${rule type}
298 [Documentation] Adds a rule, usually inserting or deleting an entry between two controllers.
299 ${base string} Set Variable sudo iptables ${rule type} OUTPUT -p all --source
300 ${cmd string} Catenate ${base string} ${isolated controller} --destination ${controller} -j DROP
301 Run Command On Remote System ${isolated controller} ${cmd string}
302 ${cmd string} Catenate ${base string} ${controller} --destination ${isolated controller} -j DROP
303 Run Command On Remote System ${isolated controller} ${cmd string}
304 ${cmd string} Set Variable sudo iptables -L -n
305 ${return string}= Run Command On Remote System ${isolated controller} ${cmd string}
307 Run Keyword If "${rule type}" == '-I' Should Match Regexp ${return string} [\s\S]*DROP *all *-- *${isolated controller} *${controller}[\s\S]*
308 Run Keyword If "${rule type}" == '-I' Should Match Regexp ${return string} [\s\S]*DROP *all *-- *${controller} *${isolated controller}[\s\S]*
310 Run Keyword If "${rule type}" == '-D' Should Match Regexp ${return string} (?![\s\S]*DROP *all *-- *${isolated controller} *${controller}[\s\S]*)
311 Run Keyword If "${rule type}" == '-D' Should Match Regexp ${return string} (?![\s\S]*DROP *all *-- *${controller} *${isolated controller}[\s\S]*)
313 Rejoin All Isolated Controllers
314 [Arguments] @{controllers}
315 [Documentation] Wipe all IPTables rules from all controllers, thus rejoining all controllers.
316 : FOR ${isolated controller} IN @{controllers}
317 \ Flush IPTables ${isolated controller}
320 [Arguments] ${isolated controller}
321 [Documentation] This keyword is generally not called from a test case but supports a complete wipe of all rules on
323 ${cmd string} Set Variable sudo iptables -v -F
324 ${return string}= Run Command On Remote System ${isolated controller} ${cmd string}
325 Log return: ${return string}
326 Should Contain ${return string} Flushing chain `INPUT'
327 Should Contain ${return string} Flushing chain `FORWARD'
328 Should Contain ${return string} Flushing chain `OUTPUT'