Reorganize and clean up cluster keywords.
Introduce START_TIMEOUT and STOP_TIMEOUT variables for instance restart.
Use cars to determine whether application is up.
Remove unnecessary waits.
Add test case documentation.
Change test cases name to standard.
Change-Id: I819cb86c604338b7452f4378ecde82d24cde6c16
Signed-off-by: Luis Gomez <ecelgp@gmail.com>
--- /dev/null
+*** Settings ***
+Library Collections
+Library CrudLibrary.py
+Library SettingsLibrary.py
+Resource DatastoreCRUD.robot
+
+*** Keywords ***
+Add Cars And Verify
+ [Arguments] ${controller_ip} ${num_cars} ${timeout}=3s
+ [Documentation] Initializes shard and then adds the specified number of cars and performs a GET as a check.
+ ${resp} InitCar ${controller_ip} ${PORT}
+ Should Be Equal As Strings ${resp.status_code} 204
+ ${resp} AddCar ${controller_ip} ${RESTCONFPORT} ${num_cars} 204
+ Should Be Equal As Strings ${resp.status_code} 204
+ Wait Until Keyword Succeeds ${timeout} 1s Get Cars And Verify ${controller_ip} ${num_cars}
+
+Add Cars And Verify Without Init
+ [Arguments] ${controller_ip} ${num_cars} ${timeout}=3s
+ [Documentation] Adds cars to an initialized cars shard then performs a GET as a check.
+ Comment First car add may return 409, but subsequent should be 204
+ ${resp} AddCar ${controller_ip} ${RESTCONFPORT} ${num_cars} 204 409
+ Should Be Equal As Strings ${resp.status_code} 204
+ Wait Until Keyword Succeeds ${timeout} 1s Get Cars And Verify ${controller_ip} ${num_cars}
+
+Get Cars And Verify
+ [Arguments] ${controller_ip} ${num_cars}
+ [Documentation] Gets cars and verifies that the manufacturer is correct.
+ # TODO: Future enhanement: verify all fields.
+ ${resp} Getcars ${controller_ip} ${RESTCONFPORT} ${0}
+ Should Be Equal As Strings ${resp.status_code} 200
+ : FOR ${i} IN RANGE 1 ${num_cars}+1
+ \ Should Contain ${resp.content} manufacturer${i}
+
+Add People And Verify
+ [Arguments] ${controller_ip} ${num_people} ${timeout}=3s
+ [Documentation] Note: The first AddPerson call passed with 0 posts directly to the data store to get
+ ... the people container created so the subsequent AddPerson RPC calls that put to the
+ ... person list will succeed.
+ ${resp} AddPerson ${controller_ip} ${RESTCONFPORT} ${0} 204
+ Should Be Equal As Strings ${resp.status_code} 204
+ Wait Until Keyword Succeeds ${timeout} 1s Get One Person And Verify ${controller_ip} ${0}
+ ${resp} AddPerson ${controller_ip} ${RESTCONFPORT} ${num_people} 200
+ Wait Until Keyword Succeeds ${timeout} 1s Get People And Verify ${controller_ip} ${num_people}
+
+Add People And Verify Without Init
+ [Arguments] ${controller_ip} ${num_people} ${timeout}=3s
+ [Documentation] Adds people to an initialized people shard then performs a GET as a check.
+ ${resp} AddPerson ${controller_ip} ${RESTCONFPORT} ${num_people} 200
+ Should Be Equal As Strings ${resp.status_code} 200
+ Wait Until Keyword Succeeds ${timeout} 1s Get People And Verify ${controller_ip} ${num_people}
+
+Get One Person And Verify
+ [Arguments] ${controller_ip} ${number}
+ [Documentation] Gets a person and verifies that the user ID is correct.
+ # TODO: Future enhanement: verify all fields.
+ ${resp} GetPersons ${controller_ip} ${RESTCONFPORT} ${0}
+ Should Be Equal As Strings ${resp.status_code} 200
+ Should Contain ${resp.content} user${number}
+
+Get People And Verify
+ [Arguments] ${controller_ip} ${num_people}
+ [Documentation] Gets multiple people and verifies that the user IDs are correct.
+ # TODO: Future enhanement: verify all fields.
+ ${resp} GetPersons ${controller_ip} ${RESTCONFPORT} ${0}
+ Should Be Equal As Strings ${resp.status_code} 200
+ : FOR ${i} IN RANGE 1 ${num_people}+1
+ \ Should Contain ${resp.content} user${i}
+
+Add Car Person And Verify
+ [Arguments] ${controller_ip} ${timeout}=3s
+ [Documentation] Add a car-person via the data store and get the car-person from Leader.
+ ... Note: This is done to get the car-people container created so subsequent
+ ... BuyCar RPC puts to the car-person list will succeed.
+ AddCarPerson ${controller_ip} ${RESTCONFPORT} ${0}
+ Wait Until Keyword Succeeds ${timeout} 1s Get One Car-Person Mapping And Verify ${controller_ip} ${0}
+
+Get One Car-Person Mapping And Verify
+ [Arguments] ${controller_ip} ${number}
+ [Documentation] Gets a car person mapping and verifies that the user ID is correct.
+ ${resp} GetCarPersonMappings ${controller_ip} ${RESTCONFPORT} ${0}
+ Should Be Equal As Strings ${resp.status_code} 200
+ Should Contain ${resp.content} user${number}
+
+Get Car-Person Mappings And Verify
+ [Arguments] ${controller_ip} ${num_entries}
+ ${resp} GetCarPersonMappings ${controller_ip} ${RESTCONFPORT} ${0}
+ Should Be Equal As Strings ${resp.status_code} 200
+ : FOR ${i} IN RANGE 1 ${num_entries}+1
+ \ Should Contain ${resp.content} user${i}
+
+Buy Cars And Verify
+ [Arguments] ${controller_ip} ${num_entries} ${start}=${0} ${timeout}=3s
+ BuyCar ${controller_ip} ${RESTCONFPORT} ${num_entries} ${start}
+ ${total_entries} Evaluate ${start}+${num_entries}
+ Wait Until Keyword Succeeds ${timeout} 1s Get Car-Person Mappings And Verify ${controller_ip} ${total_entries}
+
+Check Cars Deleted
+ [Arguments] ${controller_ip}
+ ${resp} Getcars ${controller_ip} ${RESTCONFPORT} ${0}
+ Should Be Equal As Strings ${resp.status_code} 404
+
+Delete All Cars And Verify
+ [Arguments] ${controller_ip} ${timeout}=3s
+ DeleteAllCars ${controller_ip} ${RESTCONFPORT} ${0}
+ Wait Until Keyword Succeeds ${timeout} 1s Check Cars Deleted ${controller_ip}
+
+Check People Deleted
+ [Arguments] ${controller_ip}
+ ${resp} GetPersons ${controller_ip} ${RESTCONFPORT} ${0}
+ Should Be Equal As Strings ${resp.status_code} 404
+
+Delete All People And Verify
+ [Arguments] ${controller_ip} ${timeout}=3s
+ DeleteAllPersons ${controller_ip} ${RESTCONFPORT} ${0}
+ Wait Until Keyword Succeeds ${timeout} 1s Check People Deleted ${controller_ip}
+
+Check Cars-Persons Deleted
+ [Arguments] ${controller_ip}
+ ${resp} GetCarPersonMappings ${controller_ip} ${RESTCONFPORT} ${0}
+ Should Be Equal As Strings ${resp.status_code} 404
+
+Delete All Cars-Persons And Verify
+ [Arguments] ${controller_ip} ${timeout}=3s
+ DeleteAllCarsPersons ${controller_ip} ${RESTCONFPORT} ${0}
+ Wait Until Keyword Succeeds ${timeout} 1s Check Cars-Persons Deleted ${controller_ip}
+
+Delete All Entries From Shards
+ [Documentation] Delete All Shards.
+ [Arguments] @{controllers}
+ : FOR ${ip} IN @{controllers}
+ \ Delete All Cars And Verify ${ip}
+ : FOR ${ip} IN @{controllers}
+ \ Delete All People And Verify ${ip}
+ : FOR ${ip} IN @{controllers}
+ \ Delete All Cars-Persons And Verify ${ip}
+
+Check Cars
+ [Arguments] ${controller_ip} ${nth_car}
+ [Documentation] Verifies that the first through nth car is present.
+ ${resp} Getcars ${controller_ip} ${RESTCONFPORT} ${0}
+ Should Be Equal As Strings ${resp.status_code} 200
+ : FOR ${INDEX} IN RANGE 1 ${nth_car}
+ \ ${counter}= Convert to String ${INDEX}
+ \ Log manufacturer${counter}
+ \ Should Contain ${resp.content} manufacturer${counter}
+
+Check People
+ [Arguments] ${controller_ip} ${nth_person}
+ [Documentation] Verifies that the first through nth person is present.
+ ${resp} GetPersons ${controller_ip} ${RESTCONFPORT} ${0}
+ Should Be Equal As Strings ${resp.status_code} 200
+ : FOR ${INDEX} IN RANGE 1 ${nth_person}
+ \ ${counter}= Convert to String ${INDEX}
+ \ Log user${counter}
+ \ Should Contain ${resp.content} user${counter}
+
+Check CarPeople
+ [Arguments] ${controller_ip} ${nth_carperson}
+ [Documentation] Verifies that the first through nth car-person is present.
+ ${resp} GetCarPersonMappings ${controller_ip} ${RESTCONFPORT} ${0}
+ Should Be Equal As Strings ${resp.status_code} 200
+ : FOR ${INDEX} IN RANGE 1 ${nth_carperson}
+ \ ${counter}= Convert to String ${INDEX}
+ \ Log user${counter}
+ \ Should Contain ${resp.content} user${counter}
+
+Check Elements In Shards
+ [Arguments] ${controller_ip} ${nth}
+ [Documentation] Check all shards for nth elements
+ wait until keyword succeeds 3 1 Check Cars ${controller_ip} ${nth}
+ wait until keyword succeeds 3 1 Check People ${controller_ip} ${nth}
+ wait until keyword succeeds 3 1 Check CarPeople ${controller_ip} ${nth}
+
+Initialize Cars
+ [Arguments] ${controller_ip} ${field bases}
+ [Documentation] Initializes the cars shard by creating a 0th car with POST then deleting it.
+ ... Field bases are a dictionary of datastore record field values onto which is appended
+ ... an incremental value to uniquely identify the record from which it came.
+ ... Typically, you will use the Create Dictionary keyword on arguments which look like this:
+ ... id=${EMPTY} category=coupe model=model manufacturer=mfg year=2
+ ${node}= Set Variable ${EMPTY}
+ ${prefix}= Set Variable {"car:cars":{"car-entry":[{
+ ${postfix}= Set Variable }]}}
+ Create Records ${controller_ip} ${node} ${0} ${0} ${prefix} ${field bases}
+ ... ${postfix}
+ ${node}= Set Variable car:cars/car-entry
+ Delete Records ${controller_ip} ${node} ${0} ${0}
+
+Create Cars
+ [Arguments] ${controller_ip} ${first} ${last} ${field bases}
+ [Documentation] Creates cars with record IDs of specified range using POST.
+ ... If first and last are equal, only one record is updated.
+ ... Field bases are a dictionary of datastore record field values onto which is appended
+ ... an incremental value to uniquely identify the record from which it came.
+ ... Typically, you will use the Create Dictionary keyword on an argument which looks like this:
+ ... id=${EMPTY} category=coupe model=model manufacturer=mfg year=2
+ ${node}= Set Variable car:cars
+ ${prefix}= Set Variable {"car-entry":[{
+ ${postfix}= Set Variable }]}
+ Create Records ${controller_ip} ${node} ${first} ${last} ${prefix} ${field bases}
+ ... ${postfix}
+
+Update Cars
+ [Arguments] ${controller_ip} ${first} ${last} ${field bases}
+ [Documentation] Updates cars with record IDs of the specified using PUT.
+ ... If first and last are equal, only one record is updated.
+ ... Field bases are a dictionary of datastore record field values onto which is appended
+ ... an incremental value to uniquely identify the record from which it came.
+ ... Typically, you will use the Create Dictionary keyword on arguments which look like this:
+ ... id=${EMPTY} category=coupe model=model manufacturer=mfg year=2
+ ${node}= Set Variable car:cars/car-entry
+ ${prefix}= Set Variable {"car-entry":[{
+ ${postfix}= Set Variable }]}
+ Update Records ${controller_ip} ${node} ${first} ${last} ${prefix} ${field bases}
+ ... ${postfix}
+
+Read All Cars
+ [Arguments] ${controller_ip}
+ [Documentation] Returns all records from the cars shard in JSON format.
+ ${node}= Set Variable car:cars
+ ${result}= Read Records ${controller_ip} ${node}
+ [Return] ${result}
+
+Read One Car
+ [Arguments] ${controller_ip} ${id}
+ [Documentation] Returns the specified record from the cars shard in JSON format.
+ ${node}= Set Variable car:cars/car-entry/${id}
+ ${result}= Read Records ${controller_ip} ${node}
+ [Return] ${result}
+
+Remove All Cars
+ [Arguments] ${controller_ip}
+ [Documentation] Deletes all records from the cars shard.
+ ${node}= Set Variable car:cars
+ Delete All Records ${controller_ip} ${node}
+
+Remove Cars
+ [Arguments] ${controller_ip} ${first} ${last}
+ [Documentation] Deletes the specified range of records from the cars shard.
+ ${node}= Set Variable car:cars/car-entry
+ Delete Records ${controller_ip} ${node} ${first} ${last}
+++ /dev/null
-*** Settings ***
-Library Collections
-Resource DatastoreCRUD.robot
-
-*** Keywords ***
-Initialize Cars
- [Arguments] ${controller_ip} ${field bases}
- [Documentation] Initializes the cars shard by creating a 0th car with POST then deleting it.
- ... Field bases are a dictionary of datastore record field values onto which is appended
- ... an incremental value to uniquely identify the record from which it came.
- ... Typically, you will use the Create Dictionary keyword on arguments which look like this:
- ... id=${EMPTY} category=coupe model=model manufacturer=mfg year=2
- ${node}= Set Variable ${EMPTY}
- ${prefix}= Set Variable {"car:cars":{"car-entry":[{
- ${postfix}= Set Variable }]}}
- Create Records ${controller_ip} ${node} ${0} ${0} ${prefix} ${field bases}
- ... ${postfix}
- ${node}= Set Variable car:cars/car-entry
- Delete Records ${controller_ip} ${node} ${0} ${0}
-
-Create Cars
- [Arguments] ${controller_ip} ${first} ${last} ${field bases}
- [Documentation] Creates cars with record IDs of specified range using POST.
- ... If first and last are equal, only one record is updated.
- ... Field bases are a dictionary of datastore record field values onto which is appended
- ... an incremental value to uniquely identify the record from which it came.
- ... Typically, you will use the Create Dictionary keyword on an argument which looks like this:
- ... id=${EMPTY} category=coupe model=model manufacturer=mfg year=2
- ${node}= Set Variable car:cars
- ${prefix}= Set Variable {"car-entry":[{
- ${postfix}= Set Variable }]}
- Create Records ${controller_ip} ${node} ${first} ${last} ${prefix} ${field bases}
- ... ${postfix}
-
-Update Cars
- [Arguments] ${controller_ip} ${first} ${last} ${field bases}
- [Documentation] Updates cars with record IDs of the specified using PUT.
- ... If first and last are equal, only one record is updated.
- ... Field bases are a dictionary of datastore record field values onto which is appended
- ... an incremental value to uniquely identify the record from which it came.
- ... Typically, you will use the Create Dictionary keyword on arguments which look like this:
- ... id=${EMPTY} category=coupe model=model manufacturer=mfg year=2
- ${node}= Set Variable car:cars/car-entry
- ${prefix}= Set Variable {"car-entry":[{
- ${postfix}= Set Variable }]}
- Update Records ${controller_ip} ${node} ${first} ${last} ${prefix} ${field bases}
- ... ${postfix}
-
-Read All Cars
- [Arguments] ${controller_ip}
- [Documentation] Returns all records from the cars shard in JSON format.
- ${node}= Set Variable car:cars
- ${result}= Read Records ${controller_ip} ${node}
- [Return] ${result}
-
-Read One Car
- [Arguments] ${controller_ip} ${id}
- [Documentation] Returns the specified record from the cars shard in JSON format.
- ${node}= Set Variable car:cars/car-entry/${id}
- ${result}= Read Records ${controller_ip} ${node}
- [Return] ${result}
-
-Delete All Cars
- [Arguments] ${controller_ip}
- [Documentation] Deletes all records from the cars shard.
- ${node}= Set Variable car:cars
- Delete All Records ${controller_ip} ${node}
-
-Delete Cars
- [Arguments] ${controller_ip} ${first} ${last}
- [Documentation] Deletes the specified range of records from the cars shard.
- ${node}= Set Variable car:cars/car-entry
- Delete Records ${controller_ip} ${node} ${first} ${last}
*** Settings ***
-Resource Utils.robot
Library RequestsLibrary
Library Collections
+Library UtilLibrary.py
Library ClusterStateLibrary.py
+Resource Utils.robot
*** Variables ***
${smc_node} /org.opendaylight.controller:Category=ShardManager,name=shard-manager-config,type=DistributedConfigDatastore
[Documentation] Returns the IP addr or hostname of the leader of the specified shard.
... Controllers are specifed in the pybot command line.
${searchlist} Get Controller List ${old_leader}
- ${leader} GetLeader ${shard_name} ${3} ${3} ${1} ${PORT}
+ ${leader} GetLeader ${shard_name} ${3} ${3} ${1} ${RESTCONFPORT}
... @{searchlist}
Should Not Be Equal As Strings ${leader} None
Run Keyword If '${old_leader}'!='${EMPTY}' Should Not Be Equal ${old_leader} ${leader}
[Return] ${leader}
-Wait For Leader To Be Found
+Expect No Leader
+ [Documentation] No leader is elected in the car shard
[Arguments] ${shard_name}
- [Documentation] Waits until the leader of the specified shard is found.
- ${leader} Wait Until Keyword Succeeds 12s 2s Get Leader And Verify ${shard_name}
- Log ${leader}
- [Return] ${leader}
-
-Switch Leader
- [Arguments] ${shard_name} ${current_leader}
- [Documentation] Forces a change of leadership by shutting down the current leader.
- Stop One Or More Controllers ${current_leader}
- ${new_leader} Wait Until Keyword Succeeds 60s 2s Get Leader And Verify ${shard_name} ${current_leader}
- # TODO: Future enhanement: make sure the other controller is a follower and not a master or candidate.
- Log ${new_leader}
- [Return] ${new_leader}
+ ${leader} GetLeader ${shard_name} ${3} ${1} ${1} ${RESTCONFPORT}
+ ... ${CURRENT_CAR_LEADER}
+ Should Be Equal As Strings ${leader} None
Get All Followers
[Arguments] ${shard_name} ${exclude_controller}=${EMPTY}
[Documentation] Returns the IP addresses or hostnames of all followers of the specified shard.
${searchlist} Get Controller List ${exclude_controller}
- ${followers} GetFollowers ${shard_name} ${3} ${3} ${1} ${PORT}
+ ${followers} GetFollowers ${shard_name} ${3} ${3} ${1} ${RESTCONFPORT}
... @{searchlist}
Log ${followers}
Should Not Be Empty ${followers}
[Return] ${followers}
-Add Cars And Verify
- [Arguments] ${controller_ip} ${num_cars} ${timeout}=12s
- [Documentation] Initializes shard and then adds the specified number of cars and performs a GET as a check.
- ${resp} InitCar ${controller_ip} ${PORT}
- Should Be Equal As Strings ${resp.status_code} 204
- ${resp} AddCar ${controller_ip} ${RESTCONFPORT} ${num_cars} 204
- Should Be Equal As Strings ${resp.status_code} 204
- Wait Until Keyword Succeeds ${timeout} 2s Get Cars And Verify ${controller_ip} ${num_cars}
-
-Add Cars And Verify Without Init
- [Arguments] ${controller_ip} ${num_cars} ${timeout}=12s
- [Documentation] Adds cars to an initialized cars shard then performs a GET as a check.
- Comment First car add may return 409, but subsequent should be 204
- ${resp} AddCar ${controller_ip} ${RESTCONFPORT} ${num_cars} 204 409
- Should Be Equal As Strings ${resp.status_code} 204
- Wait Until Keyword Succeeds ${timeout} 2s Get Cars And Verify ${controller_ip} ${num_cars}
-
-Get Cars And Verify
- [Arguments] ${controller_ip} ${num_cars}
- [Documentation] Gets cars and verifies that the manufacturer is correct.
- # TODO: Future enhanement: verify all fields.
- ${resp} Getcars ${controller_ip} ${RESTCONFPORT} ${0}
- Should Be Equal As Strings ${resp.status_code} 200
- : FOR ${i} IN RANGE 1 ${num_cars}+1
- \ Should Contain ${resp.content} manufacturer${i}
-
-Add People And Verify
- [Arguments] ${controller_ip} ${num_people}
- [Documentation] Note: The first AddPerson call passed with 0 posts directly to the data store to get
- ... the people container created so the subsequent AddPerson RPC calls that put to the
- ... person list will succeed.
- ${resp} AddPerson ${controller_ip} ${RESTCONFPORT} ${0} 204
- Should Be Equal As Strings ${resp.status_code} 204
- Wait Until Keyword Succeeds 12s 2s Get One Person And Verify ${controller_ip} ${0}
- ${resp} AddPerson ${controller_ip} ${RESTCONFPORT} ${num_people} 200
- Wait Until Keyword Succeeds 12s 2s Get People And Verify ${controller_ip} ${num_people}
-
-Get One Person And Verify
- [Arguments] ${controller_ip} ${number}
- [Documentation] Gets a person and verifies that the user ID is correct.
- # TODO: Future enhanement: verify all fields.
- ${resp} GetPersons ${controller_ip} ${RESTCONFPORT} ${0}
- Should Be Equal As Strings ${resp.status_code} 200
- Should Contain ${resp.content} user${number}
-
-Get People And Verify
- [Arguments] ${controller_ip} ${num_people}
- [Documentation] Gets multiple people and verifies that the user IDs are correct.
- # TODO: Future enhanement: verify all fields.
- ${resp} GetPersons ${controller_ip} ${RESTCONFPORT} ${0}
- Should Be Equal As Strings ${resp.status_code} 200
- : FOR ${i} IN RANGE 1 ${num_people}+1
- \ Should Contain ${resp.content} user${i}
-
-Add Car Person And Verify
- [Arguments] ${controller_ip}
- [Documentation] Add a car-person via the data store and get the car-person from Leader.
- ... Note: This is done to get the car-people container created so subsequent
- ... BuyCar RPC puts to the car-person list will succeed.
- AddCarPerson ${controller_ip} ${RESTCONFPORT} ${0}
- Wait Until Keyword Succeeds 60s 2s Get One Car-Person Mapping And Verify ${controller_ip} ${0}
-
-Get One Car-Person Mapping And Verify
- [Arguments] ${controller_ip} ${number}
- [Documentation] Gets a car person mapping and verifies that the user ID is correct.
- ${resp} GetCarPersonMappings ${controller_ip} ${RESTCONFPORT} ${0}
- Should Be Equal As Strings ${resp.status_code} 200
- Should Contain ${resp.content} user${number}
-
-Get Car-Person Mappings And Verify
- [Arguments] ${controller_ip} ${num_entries}
- ${resp} GetCarPersonMappings ${controller_ip} ${RESTCONFPORT} ${0}
- Should Be Equal As Strings ${resp.status_code} 200
- : FOR ${i} IN RANGE 1 ${num_entries}+1
- \ Should Contain ${resp.content} user${i}
-
-Buy Cars And Verify
- [Arguments] ${controller_ip} ${num_entries} ${start}=${0}
- Wait Until Keyword Succeeds 60s 2s BuyCar ${controller_ip} ${RESTCONFPORT} ${num_entries}
- ... ${start}
-
-Check Cars Deleted
- [Arguments] ${controller_ip}
- ${resp} Getcars ${controller_ip} ${RESTCONFPORT} ${0}
- Should Be Equal As Strings ${resp.status_code} 404
-
-Delete All Cars And Verify
- [Arguments] ${controller_ip}
- DeleteAllCars ${controller_ip} ${RESTCONFPORT} ${0}
- Wait Until Keyword Succeeds 60s 2s Check Cars Deleted ${controller_ip}
-
-Check People Deleted
- [Arguments] ${controller_ip}
- ${resp} GetPersons ${controller_ip} ${RESTCONFPORT} ${0}
- Should Be Equal As Strings ${resp.status_code} 404
-
-Delete All People And Verify
- [Arguments] ${controller_ip}
- DeleteAllPersons ${controller_ip} ${RESTCONFPORT} ${0}
- Wait Until Keyword Succeeds 60s 2s Check People Deleted ${controller_ip}
-
-Check Cars-Persons Deleted
- [Arguments] ${controller_ip}
- ${resp} GetCarPersonMappings ${controller_ip} ${RESTCONFPORT} ${0}
- Should Be Equal As Strings ${resp.status_code} 404
-
-Delete All Cars-Persons And Verify
- [Arguments] ${controller_ip}
- DeleteAllCarsPersons ${controller_ip} ${RESTCONFPORT} ${0}
- Wait Until Keyword Succeeds 60s 2s Check Cars-Persons Deleted ${controller_ip}
-
Stop One Or More Controllers
[Arguments] @{controllers}
[Documentation] Give this keyword a scalar or list of controllers to be stopped.
${cmd} = Set Variable ${KARAF_HOME}/bin/stop
: FOR ${ip} IN @{controllers}
\ Run Command On Remote System ${ip} ${cmd}
- : FOR ${ip} IN @{controllers}
- \ Wait Until Keyword Succeeds ${CONTROLLER_STOP_TIMEOUT} s 3 s Controller Down Check ${ip}
-
-Start One Or More Controllers
- [Arguments] @{controllers}
- [Documentation] Give this keyword a scalar or list of controllers to be started.
- ${cmd} = Set Variable ${KARAF_HOME}/bin/start
- : FOR ${ip} IN @{controllers}
- \ Run Command On Remote System ${ip} ${cmd}
- # TODO: This should throw an error if controller never comes up.
- : FOR ${ip} IN @{controllers}
- \ UtilLibrary.Wait For Controller Up ${ip} ${RESTCONFPORT}
Kill One Or More Controllers
[Arguments] @{controllers}
log ${cmd}
: FOR ${ip} IN @{controllers}
\ Run Command On Remote System ${ip} ${cmd}
+
+Wait For Cluster Down
+ [Arguments] ${timeout} @{controllers}
+ [Documentation] Waits for one or more clustered controllers to be down.
: FOR ${ip} IN @{controllers}
- \ Wait Until Keyword Succeeds 12 s 3 s Controller Down Check ${ip}
+ \ ${status}= Run Keyword And Return Status Wait For Controller Down ${timeout} ${ip}
+ \ Exit For Loop If '${status}' == 'FAIL'
+
+Wait For Controller Down
+ [Arguments] ${timeout} ${ip}
+ [Documentation] Waits for one controllers to be down.
+ Wait Until Keyword Succeeds ${timeout} 2s Controller Down Check ${ip}
Controller Down Check
[Arguments] ${ip}
Log Number of controller instances running: ${response}
Should Start With ${response} 0 Controller process found or there may be extra instances of karaf running on the host machine.
+Start One Or More Controllers
+ [Arguments] @{controllers}
+ [Documentation] Give this keyword a scalar or list of controllers to be started.
+ ${cmd} = Set Variable ${KARAF_HOME}/bin/start
+ : FOR ${ip} IN @{controllers}
+ \ Run Command On Remote System ${ip} ${cmd}
+
+Wait For Cluster Sync
+ [Arguments] ${timeout} @{controllers}
+ [Documentation] Waits for one or more clustered controllers to report Sync Status as true.
+ : FOR ${ip} IN @{controllers}
+ \ ${status}= Run Keyword And Return Status Wait For Controller Sync ${timeout} ${ip}
+ \ Exit For Loop If '${status}' == 'FAIL'
+
+Wait For Controller Sync
+ [Arguments] ${timeout} ${ip}
+ [Documentation] Waits for one controllers to report Sync Status as true.
+ Wait Until Keyword Succeeds ${timeout} 2s Controller Sync Status Should Be True ${ip}
+
+Controller Sync Status Should Be True
+ [Arguments] ${ip}
+ [Documentation] Checks if Sync Status is true.
+ ${SyncStatus}= Get Controller Sync Status ${ip}
+ Should Be Equal ${SyncStatus} ${True}
+
+Controller Sync Status Should Be False
+ [Arguments] ${ip}
+ [Documentation] Checks if Sync Status is false.
+ ${SyncStatus}= Get Controller Sync Status ${ip}
+ Should Be Equal ${SyncStatus} ${False}
+
+Get Controller Sync Status
+ [Arguments] ${controller_ip}
+ [Documentation] Return Sync Status.
+ ${api} Set Variable /jolokia/read
+ Create_Session session http://${controller_ip}:${RESTCONFPORT}${api} headers=${HEADERS} auth=${AUTH}
+ ${resp}= RequestsLibrary.Get session ${smc_node}
+ Log ${resp.json()}
+ Log ${resp.content}
+ ${json}= Set Variable ${resp.json()}
+ ${value}= Get From Dictionary ${json} value
+ Log value: ${value}
+ ${SyncStatus}= Get From Dictionary ${value} SyncStatus
+ Log SyncSatus: ${SyncStatus}
+ [Return] ${SyncStatus}
+
Clean One Or More Journals
[Arguments] @{controllers}
[Documentation] Give this keyword a scalar or list of controllers on which to clean journals.
: FOR ${ip} IN @{controllers}
\ Run Command On Remote System ${ip} ${cmd}
-Check Cars
- [Arguments] ${selected controller} ${PORT} ${nth car}
- [Documentation] Verifies that the first through nth car is present.
- ${resp} Getcars ${selected controller} ${PORT} ${0}
- Should Be Equal As Strings ${resp.status_code} 200
- : FOR ${INDEX} IN RANGE 1 ${nth car}
- \ ${counter}= Convert to String ${INDEX}
- \ Log manufacturer${counter}
- \ Should Contain ${resp.content} manufacturer${counter}
-
-Check People
- [Arguments] ${selected controller} ${PORT} ${nth person}
- [Documentation] Verifies that the first through nth person is present.
- ${resp} GetPersons ${selected controller} ${PORT} ${0}
- Should Be Equal As Strings ${resp.status_code} 200
- : FOR ${INDEX} IN RANGE 1 ${nth person}
- \ ${counter}= Convert to String ${INDEX}
- \ Log user${counter}
- \ Should Contain ${resp.content} user${counter}
-
-Check CarPeople
- [Arguments] ${selected controller} ${PORT} ${nth carperson}
- [Documentation] Verifies that the first through nth car-person is present.
- ${resp} GetCarPersonMappings ${selected controller} ${PORT} ${0}
- Should Be Equal As Strings ${resp.status_code} 200
- : FOR ${INDEX} IN RANGE 1 ${nth carperson}
- \ ${counter}= Convert to String ${INDEX}
- \ Log user${counter}
- \ Should Contain ${resp.content} user${counter}
-
Isolate a Controller From Cluster
[Arguments] ${isolated controller} @{controllers}
[Documentation] Use IPTables to isolate one controller from the cluster.
Should Contain ${return string} Flushing chain `FORWARD'
Should Contain ${return string} Flushing chain `OUTPUT'
-Wait for Cluster Sync
- [Arguments] ${timeout} @{controllers}
- [Documentation] Waits for one or more clustered controllers to report Sync Status as true.
- : FOR ${ip} IN @{controllers}
- \ ${resp}= Wait Until Keyword Succeeds ${timeout} 2s Controller Sync Status Should Be True ${ip}
-
-Controller Sync Status Should Be True
- [Arguments] ${ip}
- [Documentation] Checks if Sync Status is true.
- ${SyncStatus}= Get Controller Sync Status ${ip}
- Should Be Equal ${SyncStatus} ${True}
-
-Controller Sync Status Should Be False
- [Arguments] ${ip}
- [Documentation] Checks if Sync Status is false.
- ${SyncStatus}= Get Controller Sync Status ${ip}
- Should Be Equal ${SyncStatus} ${False}
-
-Get Controller Sync Status
- [Arguments] ${controller_ip}
- [Documentation] Return Sync Status.
- ${api} Set Variable /jolokia/read
- Create_Session session http://${controller_ip}:${RESTCONFPORT}${api} headers=${HEADERS} auth=${AUTH}
- ${resp}= RequestsLibrary.Get session ${smc_node}
- Log ${resp.json()}
- Log ${resp.content}
- ${json}= Set Variable ${resp.json()}
- ${value}= Get From Dictionary ${json} value
- Log value: ${value}
- ${SyncStatus}= Get From Dictionary ${value} SyncStatus
- Log SyncSatus: ${SyncStatus}
- [Return] ${SyncStatus}
*** Variables ***
@{controllers} ${CONTROLLER} ${CONTROLLER1} ${CONTROLLER2}
${KARAF_HOME} ${WORKSPACE}/${BUNDLEFOLDER}
+${START_TIMEOUT} 300s
+${STOP_TIMEOUT} 180s
*** Test Cases ***
Stop All Controllers
[Documentation] Stop all the controllers in the cluster.
Stop One Or More Controllers @{controllers}
+ Wait For Cluster Down ${STOP_TIMEOUT} @{controllers}
Clean All Journals
[Documentation] Clean the journals of all the controllers in the cluster
Start All Controllers
[Documentation] Start all the controllers in the cluster
Start One Or More Controllers @{controllers}
+ Wait For Cluster Sync ${START_TIMEOUT} @{controllers}
*** Settings ***
Documentation This test finds the leader for shards in a 3-Node cluster and executes CRUD operations on them
Default Tags 3-node-cluster
-Library Collections
-Library RequestsLibrary
-Library ../../../libraries/Common.py
-Library ../../../libraries/CrudLibrary.py
-Library ../../../libraries/SettingsLibrary.py
-Library ../../../libraries/UtilLibrary.py
-Library ../../../libraries/ClusterStateLibrary.py
Resource ../../../libraries/ClusterKeywords.robot
+Resource ../../../libraries/CarsAndPeople.robot
Variables ../../../variables/Variables.py
*** Variables ***
*** Test Cases ***
Get Car Leader And Followers
- ${CURRENT_CAR_LEADER} Wait For Leader To Be Found ${SHARD_CAR_NAME}
+ [Documentation] Find leader and followers in the car shard
+ ${CURRENT_CAR_LEADER} Get Leader And Verify ${SHARD_CAR_NAME}
Set Suite Variable ${CURRENT_CAR_LEADER}
${CAR_FOLLOWERS} Get All Followers ${SHARD_CAR_NAME}
Set Suite Variable ${CAR_FOLLOWERS}
-Add cars and get cars from Leader
+Add Cars And Get Cars From Leader
[Documentation] Add some cars and get added cars from Leader
Add Cars And Verify ${CURRENT_CAR_LEADER} ${NUM_ENTRIES}
-Get added cars from Follower1
+Get Added Cars From Follower1
[Documentation] Get added cars from Follower1
- Wait Until Keyword Succeeds 60s 2s Get Cars And Verify @{CAR_FOLLOWERS}[0] ${NUM_ENTRIES}
+ Get Cars And Verify @{CAR_FOLLOWERS}[0] ${NUM_ENTRIES}
-Get added cars from Follower2
+Get Added Cars From Follower2
[Documentation] Get added cars from Follower2
- Wait Until Keyword Succeeds 60s 2s Get Cars And Verify @{CAR_FOLLOWERS}[1] ${NUM_ENTRIES}
+ Get Cars And Verify @{CAR_FOLLOWERS}[1] ${NUM_ENTRIES}
Get People Leader And Followers
- ${CURRENT_PEOPLE_LEADER} Wait For Leader To Be Found ${SHARD_PEOPLE_NAME}
+ [Documentation] Find leader and followers in the people shard
+ ${CURRENT_PEOPLE_LEADER} Get Leader And Verify ${SHARD_PEOPLE_NAME}
Set Suite Variable ${CURRENT_PEOPLE_LEADER}
${PEOPLE_FOLLOWERS} Get All Followers ${SHARD_PEOPLE_NAME}
Set Suite Variable ${PEOPLE_FOLLOWERS}
-Add people and get people from Leader
+Add People And Get People From Leader
[Documentation] Add some people and get people from Leader.
Add People And Verify ${CURRENT_PEOPLE_LEADER} ${NUM_ENTRIES}
-Get added people from Follower1
+Get Added People From Follower1
[Documentation] Get added people from Follower1
- Wait Until Keyword Succeeds 60s 2s Get People And Verify @{PEOPLE_FOLLOWERS}[0] ${NUM_ENTRIES}
+ Get People And Verify @{PEOPLE_FOLLOWERS}[0] ${NUM_ENTRIES}
-Get added people from Follower2
+Get Added People From Follower2
[Documentation] Get added people from Follower2
- Wait Until Keyword Succeeds 60s 2s Get People And Verify @{PEOPLE_FOLLOWERS}[1] ${NUM_ENTRIES}
+ Get People And Verify @{PEOPLE_FOLLOWERS}[1] ${NUM_ENTRIES}
Get Car-Person Leader And Followers
- ${CURRENT_CAR_PERSON_LEADER} Wait For Leader To Be Found ${SHARD_CAR_PERSON_NAME}
+ [Documentation] Find leader and followers in the car-person shard
+ ${CURRENT_CAR_PERSON_LEADER} Get Leader And Verify ${SHARD_CAR_PERSON_NAME}
Set Suite Variable ${CURRENT_CAR_PERSON_LEADER}
${CAR_PERSON_FOLLOWERS} Get All Followers ${SHARD_CAR_PERSON_NAME}
Set Suite Variable ${CAR_PERSON_FOLLOWERS}
-Add car-person mapping and get car-person mapping from Leader
+Add Car-Person Mapping And Get Car-Person Mapping From Leader
+ [Documentation] Initialize car-person shard
Add Car Person And Verify ${CURRENT_CAR_PERSON_LEADER}
-Purchase cars on Leader
+Purchase Cars On Leader
[Documentation] Purchase some cars on the Leader
${NUM_BUY_CARS_ON_LEADER} Evaluate ${NUM_ENTRIES}/3
${NUM_BUY_CARS_ON_FOLLOWER1} Evaluate ${NUM_ENTRIES}/3
Set Suite Variable ${NUM_BUY_CARS_ON_FOLLOWER2}
Buy Cars And Verify ${CURRENT_CAR_PERSON_LEADER} ${NUM_BUY_CARS_ON_LEADER}
-Purchase cars on Follower1
+Purchase Cars On Follower1
[Documentation] Purchase some cars on Follower1
Buy Cars And Verify @{CAR_PERSON_FOLLOWERS}[0] ${NUM_BUY_CARS_ON_FOLLOWER1} ${NUM_BUY_CARS_ON_LEADER}
-Purchase cars on Follower2
+Purchase Cars On Follower2
[Documentation] Purchase some cars on Follower2
${start} Evaluate ${NUM_BUY_CARS_ON_LEADER}+${NUM_BUY_CARS_ON_FOLLOWER1}
Buy Cars And Verify @{CAR_PERSON_FOLLOWERS}[1] ${NUM_BUY_CARS_ON_FOLLOWER2} ${start}
-Get car-person mappings from Leader
+Get Car-Person Mappings From Leader
[Documentation] Get car-person mappings from Leader to see all entries
- Wait Until Keyword Succeeds 60s 2s Get Car-Person Mappings And Verify ${CURRENT_CAR_PERSON_LEADER} ${NUM_ENTRIES}
+ Get Car-Person Mappings And Verify ${CURRENT_CAR_PERSON_LEADER} ${NUM_ENTRIES}
-Get car-person mappings from Follower1
+Get Car-Person Mappings From Follower1
[Documentation] Get car-person mappings from Follower1 to see all entries
- Wait Until Keyword Succeeds 60s 2s Get Car-Person Mappings And Verify @{CAR_PERSON_FOLLOWERS}[0] ${NUM_ENTRIES}
+ Get Car-Person Mappings And Verify @{CAR_PERSON_FOLLOWERS}[0] ${NUM_ENTRIES}
-Get car-person mappings from Follower2
+Get Car-Person Mappings From Follower2
[Documentation] Get car-person mappings from Follower2 to see all entries
- Wait Until Keyword Succeeds 60s 2s Get Car-Person Mappings And Verify @{CAR_PERSON_FOLLOWERS}[1] ${NUM_ENTRIES}
+ Get Car-Person Mappings And Verify @{CAR_PERSON_FOLLOWERS}[1] ${NUM_ENTRIES}
*** Settings ***
Documentation This test finds the followers of certain shards in a 3-Node cluster and executes CRUD operations on any one follower
Default Tags 3-node-cluster
-Library Collections
-Library ../../../libraries/Common.py
-Library ../../../libraries/CrudLibrary.py
-Library ../../../libraries/SettingsLibrary.py
-Library ../../../libraries/UtilLibrary.py
-Library ../../../libraries/ClusterStateLibrary.py
Resource ../../../libraries/ClusterKeywords.robot
+Resource ../../../libraries/CarsAndPeople.robot
Variables ../../../variables/Variables.py
*** Variables ***
*** Test Cases ***
Get Car Followers
+ [Documentation] Find followers in the car shard
${CAR_FOLLOWERS} Get All Followers ${SHARD_CAR_NAME}
Set Suite Variable ${CAR_FOLLOWERS}
Get People Followers
+ [Documentation] Find followers in the people shard
${PEOPLE_FOLLOWERS} Get All Followers ${SHARD_PEOPLE_NAME}
Set Suite Variable ${PEOPLE_FOLLOWERS}
${CAR_PERSON_FOLLOWERS} Get All Followers ${SHARD_CAR_PERSON_NAME}
Set Suite Variable ${CAR_PERSON_FOLLOWERS}
-Delete cars from Follower1
+Delete Cars From Follower1
Delete All Cars And Verify @{CAR_FOLLOWERS}[0]
-Delete people from Follower1
+Delete People From Follower1
Delete All People And Verify @{PEOPLE_FOLLOWERS}[0]
-Delete car-persons from Follower1
+Delete Car-Persons from Follower1
Delete All Cars-Persons And Verify @{CAR_PERSON_FOLLOWERS}[0]
-Add cars and get cars from Follower1
+Add Cars And Get Cars From Follower1
[Documentation] Add cars and get added cars from Follower1
Add Cars And Verify @{CAR_FOLLOWERS}[0] ${NUM_ENTRIES}
-Get added cars from Follower2
+Get Added Cars From Follower2
[Documentation] Get added cars from Follower2
- Wait Until Keyword Succeeds 60s 2s Get Cars And Verify @{CAR_FOLLOWERS}[1] ${NUM_ENTRIES}
+ Get Cars And Verify @{CAR_FOLLOWERS}[1] ${NUM_ENTRIES}
-Add people and get people from Follower1
+Add People And Get People From Follower1
[Documentation] Add people and get people from Follower1
Add People And Verify @{PEOPLE_FOLLOWERS}[0] ${NUM_ENTRIES}
-Get added people from Follower2
+Get Added People From Follower2
[Documentation] Get added people from Follower2
- Wait Until Keyword Succeeds 60s 2s Get People And Verify @{PEOPLE_FOLLOWERS}[1] ${NUM_ENTRIES}
+ Get People And Verify @{PEOPLE_FOLLOWERS}[1] ${NUM_ENTRIES}
-Add car-person mapping and get car-person mapping from Follower1
+Add Car-Person Mapping And Get Car-Person Mapping From Follower1
Add Car Person And Verify @{CAR_PERSON_FOLLOWERS}[0]
-Purchase cars on Follower1
+Purchase Cars On Follower1
[Documentation] Purchase cars using Follower1
Buy Cars And Verify @{CAR_PERSON_FOLLOWERS}[0] ${NUM_ENTRIES}
-Get car-person mappings from Follower1
+Get Car-Person Mappings From Follower1
[Documentation] Get car-person mappings from Follower1 to see all entries
- Wait Until Keyword Succeeds 60s 2s Get Car-Person Mappings And Verify @{CAR_PERSON_FOLLOWERS}[0] ${NUM_ENTRIES}
+ Get Car-Person Mappings And Verify @{CAR_PERSON_FOLLOWERS}[0] ${NUM_ENTRIES}
-Get car-person mappings from Leader
+Get Car-Person Mappings From Leader
[Documentation] Get car-person mappings from the Leader to see all entries
- ${CURRENT_CAR_LEADER} Wait For Leader To Be Found ${SHARD_CAR_PERSON_NAME}
- Wait Until Keyword Succeeds 60s 2s Get Car-Person Mappings And Verify ${CURRENT_CAR_LEADER} ${NUM_ENTRIES}
+ ${CURRENT_CAR_LEADER} Get Leader And Verify ${SHARD_CAR_PERSON_NAME}
+ Get Car-Person Mappings And Verify ${CURRENT_CAR_LEADER} ${NUM_ENTRIES}
-Get car-person mappings from Follower2
+Get Car-Person Mappings From Follower2
[Documentation] Get car-person mappings from Follower2 to see all entries
- Wait Until Keyword Succeeds 60s 2s Get Car-Person Mappings And Verify @{CAR_PERSON_FOLLOWERS}[1] ${NUM_ENTRIES}
+ Get Car-Person Mappings And Verify @{CAR_PERSON_FOLLOWERS}[1] ${NUM_ENTRIES}
Documentation This test brings down the current leader of the "car" shard and then executes CRUD
... operations on the new leader
Default Tags 3-node-cluster
-Library ../../../libraries/CrudLibrary.py
-Library ../../../libraries/UtilLibrary.py
-Library ../../../libraries/ClusterStateLibrary.py
Resource ../../../libraries/ClusterKeywords.robot
+Resource ../../../libraries/CarsAndPeople.robot
+Variables ../../../variables/Variables.py
*** Variables ***
${CAR_SHARD} shard-car-config
${NUM_CARS} ${50}
${NUM_ORIG_CARS} ${10}
${KARAF_HOME} ${WORKSPACE}/${BUNDLEFOLDER}
+${START_TIMEOUT} 300s
+${STOP_TIMEOUT} 180s
*** Test Cases ***
-Get old car leader
- ${OLD_CAR_LEADER} Wait For Leader To Be Found ${CAR_SHARD}
+Get Old Car Leader
+ [Documentation] Find leader in the car shard
+ ${OLD_CAR_LEADER} Get Leader And Verify ${CAR_SHARD}
Set Suite Variable ${OLD_CAR_LEADER}
-Delete cars on old leader
+Delete Cars On Old Leader
+ [Documentation] Delete cars in Leader
Delete All Cars And Verify ${OLD_CAR_LEADER}
-Add original cars on old leader
+Add Original Cars On Old Leader
+ [Documentation] Add new cars in Leader and verify
Add Cars And Verify ${OLD_CAR_LEADER} ${NUM_ORIG_CARS}
-Switch car leader
+Switch Car Leader
[Documentation] Stop the leader to cause a new leader to be elected
- ${NEW_CAR_LEADER} Switch Leader ${CAR_SHARD} ${OLD_CAR_LEADER}
+ Stop One Or More Controllers ${OLD_CAR_LEADER}
+ Wait For Controller Down ${STOP_TIMEOUT} ${OLD_CAR_LEADER}
+ ${NEW_CAR_LEADER} Wait Until Keyword Succeeds 30s 2s Get Leader And Verify ${CAR_SHARD} ${OLD_CAR_LEADER}
Set Suite Variable ${NEW_CAR_LEADER}
-Get original cars on new leader
- Wait Until Keyword Succeeds 60s 2s Get Cars And Verify ${NEW_CAR_LEADER} ${NUM_ORIG_CARS}
+Get Original Cars On New Leader
+ [Documentation] Get cars in new Leader
+ Get Cars And Verify ${NEW_CAR_LEADER} ${NUM_ORIG_CARS}
-Delete cars on new leader
+Delete Cars On New Leader
+ [Documentation] Delete cars in new Leader
Delete All Cars And Verify ${NEW_CAR_LEADER}
-Add new cars and get cars from new leader
+Add New Cars And Get Cars From New Leader
[Documentation] Add cars and get added cars from the Leader
Add Cars And Verify ${NEW_CAR_LEADER} ${NUM_CARS}
Get Car Followers
+ [Documentation] Find followers in the car shard
${CAR_FOLLOWERS} Get All Followers ${CAR_SHARD} ${OLD_CAR_LEADER}
Set Suite Variable ${CAR_FOLLOWERS}
-Get added cars from Follower
+Get Added Cars From Follower
[Documentation] Get the added cars from the Follower
- Wait Until Keyword Succeeds 60s 2s Get Cars And Verify @{CAR_FOLLOWERS}[0] ${NUM_CARS}
+ Get Cars And Verify @{CAR_FOLLOWERS}[0] ${NUM_CARS}
-Delete cars on Follower
+Delete Cars On Follower
+ [Documentation] Delete cars in follower
Delete All Cars And Verify @{CAR_FOLLOWERS}[0]
-Add cars from Follower
+Add Cars From Follower
[Documentation] Add more cars from the Follower
Add Cars And Verify @{CAR_FOLLOWERS}[0] ${NUM_CARS}
-Get added cars from new leader
+Get Added Cars From New Leader
[Documentation] Get added cars from the new leader
- Wait Until Keyword Succeeds 60s 2s Get Cars And Verify ${NEW_CAR_LEADER} ${NUM_CARS}
+ Get Cars And Verify ${NEW_CAR_LEADER} ${NUM_CARS}
-Restart old Car leader
+Restart Old Car Leader
+ [Documentation] Start old car Leader
Start One Or More Controllers ${OLD_CAR_LEADER}
+ Wait For Controller Sync ${START_TIMEOUT} ${OLD_CAR_LEADER}
-Get added cars from old leader
+Get Added Cars From Old Leader
[Documentation] Get the added cars from the old leader
- Wait Until Keyword Succeeds 60s 2s Get Cars And Verify ${OLD_CAR_LEADER} ${NUM_CARS}
+ Wait Until Keyword Succeeds ${START_TIMEOUT} 2s Get Cars And Verify ${OLD_CAR_LEADER} ${NUM_CARS}
*** Settings ***
Documentation This test brings down the current leader of the "car" shard and then executes CRUD operations on the new leader
Default Tags 3-node-cluster
-Library ../../../libraries/CrudLibrary.py
-Library ../../../libraries/UtilLibrary.py
-Library ../../../libraries/ClusterStateLibrary.py
Resource ../../../libraries/ClusterKeywords.robot
+Resource ../../../libraries/CarsAndPeople.robot
+Variables ../../../variables/Variables.py
*** Variables ***
${PEOPLE_SHARD} shard-people-config
${NUM_ENTRIES} ${50}
${KARAF_HOME} ${WORKSPACE}/${BUNDLEFOLDER}
+${START_TIMEOUT} 300s
+${STOP_TIMEOUT} 180s
*** Test Cases ***
+Get Old People Leader
+ [Documentation] Find leader in the people shard
+ ${OLD_PEOPLE_LEADER} Get Leader And Verify ${PEOPLE_SHARD}
+ Set Suite Variable ${OLD_PEOPLE_LEADER}
+
Switch People Leader
[Documentation] Stop the leader to cause a new leader to be elected
- ${OLD_PEOPLE_LEADER} Wait For Leader To Be Found ${PEOPLE_SHARD}
- ${NEW_PEOPLE_LEADER} Switch Leader ${PEOPLE_SHARD} ${OLD_PEOPLE_LEADER}
- Set Suite Variable ${OLD_PEOPLE_LEADER}
+ Stop One Or More Controllers ${OLD_PEOPLE_LEADER}
+ Wait For Controller Down ${STOP_TIMEOUT} ${OLD_PEOPLE_LEADER}
+ ${NEW_PEOPLE_LEADER} Wait Until Keyword Succeeds 30s 2s Get Leader And Verify ${PEOPLE_SHARD} ${OLD_PEOPLE_LEADER}
Set Suite Variable ${NEW_PEOPLE_LEADER}
-Delete people from new leader
+Delete People From New Leader
+ [Documentation] Delete people in new Leader
Delete All People And Verify ${NEW_PEOPLE_LEADER}
-Add people and get from new leader
+Add People And Get From New Leader
[Documentation] Add people and get people from new leader
Add People And Verify ${NEW_PEOPLE_LEADER} ${NUM_ENTRIES}
Get People Followers
+ [Documentation] Find followers in the people shard
${PEOPLE_FOLLOWERS} Get All Followers ${PEOPLE_SHARD} ${OLD_PEOPLE_LEADER}
Set Suite Variable ${PEOPLE_FOLLOWERS}
-Get added people from Follower
- Wait Until Keyword Succeeds 60s 2s Get People And Verify @{PEOPLE_FOLLOWERS}[0] ${NUM_ENTRIES}
+Get Added People From Follower
+ [Documentation] Get people in follower and verify
+ Get People And Verify @{PEOPLE_FOLLOWERS}[0] ${NUM_ENTRIES}
-Delete people from new Follower
+Delete People From New Follower
+ [Documentation] Delete people in follower and verify
Delete All People And Verify @{PEOPLE_FOLLOWERS}[0]
-Add people from new Follower
- [Documentation] Add people and get people from follower
+Add People From New Follower
+ [Documentation] Add people in follower and verify
Add People And Verify @{PEOPLE_FOLLOWERS}[0] ${NUM_ENTRIES}
-Get added people from new leader
- Wait Until Keyword Succeeds 60s 2s Get People And Verify ${NEW_PEOPLE_LEADER} ${NUM_ENTRIES}
+Get Added People From New Leader
+ [Documentation] Get people in Leader and verify
+ Get People And Verify ${NEW_PEOPLE_LEADER} ${NUM_ENTRIES}
-Restart old People leader
+Restart Old People Leader
+ [Documentation] Start old people Leader
Start One Or More Controllers ${OLD_PEOPLE_LEADER}
+ Wait For Controller Sync ${START_TIMEOUT} ${OLD_PEOPLE_LEADER}
+
+Check Cars In Old People Leader
+ [Documentation] Check cars in new Leader. This is to avoid delay when RPC does not work.
+ Wait Until Keyword Succeeds ${START_TIMEOUT} 2s Check Cars ${OLD_PEOPLE_LEADER} ${NUM_ENTRIES}
-Get added people from old leader
- Wait Until Keyword Succeeds 60s 2s Get People And Verify ${OLD_PEOPLE_LEADER} ${NUM_ENTRIES}
+Get Added People From Old Leader
+ [Documentation] Get people in old Leader and verify
+ Get People And Verify ${OLD_PEOPLE_LEADER} ${NUM_ENTRIES}
*** Settings ***
Documentation This test restarts all controllers to verify recovery of car data from persistene
Default Tags 3-node-cluster
-Library ../../../libraries/CrudLibrary.py
-Library ../../../libraries/UtilLibrary.py
-Library ../../../libraries/ClusterStateLibrary.py
Resource ../../../libraries/ClusterKeywords.robot
+Resource ../../../libraries/CarsAndPeople.robot
Variables ../../../variables/Variables.py
*** Variables ***
${NUM_CARS} ${50}
${KARAF_HOME} ${WORKSPACE}/${BUNDLEFOLDER}
@{controllers} ${CONTROLLER} ${CONTROLLER1} ${CONTROLLER2}
+${START_TIMEOUT} 300s
+${STOP_TIMEOUT} 180s
*** Test Cases ***
-Get car leader
- ${CAR_LEADER} Wait For Leader To Be Found ${CAR_SHARD}
+Get Car Leader
+ [Documentation] Find leader in the car shard
+ ${CAR_LEADER} Get Leader And Verify ${CAR_SHARD}
Set Suite Variable ${CAR_LEADER}
-Delete cars from leader
+Delete Cars From Leader
+ [Documentation] Delete cars in Leader
Delete All Cars And Verify ${CAR_LEADER}
-Stop all controllers after delete
+Stop All Controllers After Delete
+ [Documentation] Stop all controllers
Stop One Or More Controllers @{controllers}
+ Wait For Cluster Down ${STOP_TIMEOUT} @{controllers}
-Start all controllers after delete
+Start All Controllers After Delete
+ [Documentation] Start all controller
Start One Or More Controllers @{controllers}
+ Wait For Cluster Sync ${START_TIMEOUT} @{controllers}
-Verify no cars on leader after restart
- ${resp} Getcars ${CAR_LEADER} ${RESTCONFPORT} ${0}
- Should Be Equal As Strings ${resp.status_code} 404
+Get Car Leader After First Restart
+ [Documentation] Find leader in the car shard
+ ${CAR_LEADER} Get Leader And Verify ${CAR_SHARD}
+ Set Suite Variable ${CAR_LEADER}
+
+Verify No Cars On Leader After Restart
+ [Documentation] Verify no cars after restart
+ Wait Until Keyword Succeeds ${START_TIMEOUT} 2s Check Cars Deleted ${CAR_LEADER}
-Add cars on leader
+Add Cars On Leader
+ [Documentation] Add cars in Leader
Add Cars And Verify ${CAR_LEADER} ${NUM_CARS}
-Stop all controllers after add
+Stop All Controllers After Add
+ [Documentation] Stop all controllers
Stop One Or More Controllers @{controllers}
+ Wait For Cluster Down ${STOP_TIMEOUT} @{controllers}
-Start all controllers after add
+Start All Controllers After Add
+ [Documentation] Start all controllers
Start One Or More Controllers @{controllers}
+ Wait For Cluster Sync ${START_TIMEOUT} @{controllers}
-Get cars from leader after restart
- Wait Until Keyword Succeeds 60s 2s Get Cars And Verify ${CAR_LEADER} ${NUM_CARS}
+Get Car Leader After Second Restart
+ [Documentation] Find leader in the car shard
+ ${CAR_LEADER} Get Leader And Verify ${CAR_SHARD}
+ Set Suite Variable ${CAR_LEADER}
-Get car followers
+Get Cars From Leader After Restart
+ [Documentation] Get cars from Leader and verify
+ Wait Until Keyword Succeeds ${START_TIMEOUT} 2s Get Cars And Verify ${CAR_LEADER} ${NUM_CARS}
+
+Get Car Followers
+ [Documentation] Find followers in the car shard
${CAR_FOLLOWERS} Get All Followers ${CAR_SHARD}
Set Suite Variable ${CAR_FOLLOWERS}
-Get cars from Follower1 after restart
- Wait Until Keyword Succeeds 60s 2s Get Cars And Verify @{CAR_FOLLOWERS}[0] ${NUM_CARS}
+Get Cars From Follower1 After Restart
+ [Documentation] Get cars in follower and verify
+ Wait Until Keyword Succeeds ${START_TIMEOUT} 2s Get Cars And Verify @{CAR_FOLLOWERS}[0] ${NUM_CARS}
+
+Get Cars From Follower2 After Restart
+ [Documentation] Get cars in follower and verify
+ Wait Until Keyword Succeeds ${START_TIMEOUT} 2s Get Cars And Verify @{CAR_FOLLOWERS}[1] ${NUM_CARS}
-Get cars from Follower2 after restart
- Wait Until Keyword Succeeds 60s 2s Get Cars And Verify @{CAR_FOLLOWERS}[1] ${NUM_CARS}
Default Tags 3-node-cluster
Library Collections
Library RequestsLibrary
-Library ../../../libraries/Common.py
Library ../../../libraries/CrudLibrary.py
-Library ../../../libraries/SettingsLibrary.py
-Library ../../../libraries/UtilLibrary.py
-Library ../../../libraries/ClusterStateLibrary.py
Resource ../../../libraries/ClusterKeywords.robot
+Resource ../../../libraries/CarsAndPeople.robot
Variables ../../../variables/Variables.py
*** Variables ***
${NUM_CARS} ${60}
@{controllers} ${CONTROLLER} ${CONTROLLER1} ${CONTROLLER2}
${KARAF_HOME} ${WORKSPACE}/${BUNDLEFOLDER}
+${START_TIMEOUT} 300s
+${STOP_TIMEOUT} 180s
*** Test Cases ***
Stop All Controllers
[Documentation] Stop all the controllers in the cluster
Stop One Or More Controllers @{controllers}
+ Wait For Cluster Down ${STOP_TIMEOUT} @{controllers}
Clean All Journals
[Documentation] Clean the journals of all the controllers in the cluster
Start All Controllers
[Documentation] Start all the controllers in the cluster
Start One Or More Controllers @{controllers}
+ Wait For Cluster Sync ${START_TIMEOUT} @{controllers}
-Get car leader and followers
- ${CURRENT_CAR_LEADER} Wait For Leader To Be Found ${CAR_SHARD}
+Get Car Leader And Followers
+ [Documentation] Find leader and followers in the car shard
+ ${CURRENT_CAR_LEADER} Get Leader And Verify ${CAR_SHARD}
Set Suite Variable ${CURRENT_CAR_LEADER}
${CAR_FOLLOWERS} Get All Followers ${CAR_SHARD}
Set Suite Variable ${CAR_FOLLOWERS}
-Stop both of the followers
+Verify No Cars On Leader After Restart
+ [Documentation] Verify no cars after restart
+ Wait Until Keyword Succeeds ${START_TIMEOUT} 2s Check Cars Deleted ${CURRENT_CAR_LEADER}
+
+Stop Both Of The Followers
+ [Documentation] Stop car followers
@{followers} = Create List @{CAR_FOLLOWERS}[0] @{CAR_FOLLOWERS}[1]
Stop One Or More Controllers @{followers}
+ Wait For Cluster Down ${STOP_TIMEOUT} @{followers}
-Attempt to add a car to the leader
- [Documentation] Should fail as both followers are down
+Attempt To Add A Car To The Leader
+ [Documentation] Add car should fail as both followers are down
AddCar ${CURRENT_CAR_LEADER} ${RESTCONFPORT} ${1} 500
Sleep 2
${resp} Getcars ${CURRENT_CAR_LEADER} ${RESTCONFPORT} ${1}
Should Not Be Equal As Strings ${resp.status_code} 200
-Restart the first follower
+Restart The First Follower
+ [Documentation] Start one follower
Start One Or More Controllers @{CAR_FOLLOWERS}[0]
+ Wait For Controller Sync ${START_TIMEOUT} @{CAR_FOLLOWERS}[0]
-Add cars to the first follower
+Add Cars To The First Follower
+ [Documentation] Add cars to the follower and verify
Log Adding ${NUM_CARS} cars to @{CAR_FOLLOWERS}[0]
- Add Cars And Verify Without Init @{CAR_FOLLOWERS}[0] ${NUM_CARS} 4s
+ Wait Until Keyword Succeeds ${START_TIMEOUT} 2s Add Cars And Verify Without Init @{CAR_FOLLOWERS}[0] ${NUM_CARS}
-Restart the second follower
+Restart The Second Follower
+ [Documentation] Start another follower
Start One Or More Controllers @{CAR_FOLLOWERS}[1]
+ Wait For Controller Sync ${START_TIMEOUT} @{CAR_FOLLOWERS}[1]
-Get all the cars from the second follower
- Get Cars And Verify @{CAR_FOLLOWERS}[1] ${NUM_CARS}
+Get All The Cars From The Second Follower
+ [Documentation] Add cars to the follower and verify
+ Wait Until Keyword Succeeds ${START_TIMEOUT} 2s Get Cars And Verify @{CAR_FOLLOWERS}[1] ${NUM_CARS}
-*** Keywords ***
-Expect No Leader
- ${leader} GetLeader ${CAR_SHARD} ${3} ${1} ${1} ${RESTCONFPORT}
- ... ${CURRENT_CAR_LEADER}
- Should Be Equal As Strings ${leader} None
Documentation Test suite for Routed RPC.
Library Collections
Library RequestsLibrary
-Library ../../../libraries/Common.py
Library ../../../libraries/CrudLibrary.py
-Library ../../../libraries/SettingsLibrary.py
-Library ../../../libraries/UtilLibrary.py
Resource ../../../libraries/ClusterKeywords.robot
+Resource ../../../libraries/CarsAndPeople.robot
Variables ../../../variables/Variables.py
*** Variables ***
${SHARD_PEOPLE_NAME} shard-people-config
${SHARD_CAR_PERSON_NAME} shard-car-people-config
${KARAF_HOME} ${WORKSPACE}/${BUNDLEFOLDER}
+${NUM_ENTRIES} ${100}
+${START_TIMEOUT} 300s
+${STOP_TIMEOUT} 180s
*** Test Cases ***
-Delete all entries from shards
+Delete All Entries
[Documentation] Make sure the shards are cleared for testing.
- : FOR ${ip} IN @{controllers}
- \ Delete All Cars And Verify ${ip}
- : FOR ${ip} IN @{controllers}
- \ Delete All People And Verify ${ip}
- : FOR ${ip} IN @{controllers}
- \ Delete All Cars-Persons And Verify ${ip}
+ Delete All Entries From Shards @{controllers}
Get Car Leader And Followers
- ${CURRENT_CAR_LEADER} Wait For Leader To Be Found ${SHARD_CAR_NAME}
+ [Documentation] Find leader and followers in the car shard
+ ${CURRENT_CAR_LEADER} Get Leader And Verify ${SHARD_CAR_NAME}
Set Suite Variable ${CURRENT_CAR_LEADER}
@{CAR_FOLLOWERS} Get All Followers ${SHARD_CAR_NAME}
Set Suite Variable ${CAR_FOLLOWERS}
-Add cars and get cars from Leader
+Add Cars And Get Cars From Leader
[Documentation] Add 100 cars and get added cars from Leader
- ${resp}= InitCar ${CURRENT_CAR_LEADER} ${PORT}
- ${resp}= AddCar ${CURRENT_CAR_LEADER} ${PORT} ${100}
- ${resp}= Getcars ${CURRENT_CAR_LEADER} ${PORT} ${0}
- Should Be Equal As Strings ${resp.status_code} 200
- Should Contain ${resp.content} manufacturer1 cars not added!
-
-Add persons and get persons from Leader
- [Documentation] Add 100 persons and get persons Note: There should be one person added first to enable rpc
- ${resp} AddPerson ${CURRENT_CAR_LEADER} ${PORT} ${0}
- ${resp} AddPerson ${CURRENT_CAR_LEADER} ${PORT} ${100}
- ${resp} GetPersons ${CURRENT_CAR_LEADER} ${PORT} ${0}
- Should Be Equal As Strings ${resp.status_code} 200
- Should Contain ${resp.content} user5 people not added!
-
-Add car-person mapping and get car-person mapping from Follower1
- [Documentation] Add car-person and get car-person from Leader Note: This is done to enable working of rpc
- Log @{CAR_FOLLOWERS}[0]
- ${resp} AddCarPerson @{CAR_FOLLOWERS}[0] ${PORT} ${0}
- Should Be Equal As Strings ${resp.status_code} 204
- ${resp} GetCarPersonMappings @{CAR_FOLLOWERS}[0] ${PORT} ${0}
- Should Be Equal As Strings ${resp.status_code} 200
- Should Contain ${resp.content} user0 car-person not initialized!
+ Add Cars And Verify ${CURRENT_CAR_LEADER} ${NUM_ENTRIES}
+
+Add Persons And Get Persons From Car Leader
+ [Documentation] Add 100 persons and get persons from Leader
+ Add People And Verify ${CURRENT_CAR_LEADER} ${NUM_ENTRIES}
+
+Add Car-Person Mapping And Get Car-Person Mapping From Car Follower1
+ [Documentation] Add car-person and get car-person from Leader
+ Add Car Person And Verify @{CAR_FOLLOWERS}[0]
-Purchase 100 cars using Follower1
+Purchase 100 Cars Using Car Follower1
[Documentation] Purchase 100 cars using Follower1
- BuyCar @{CAR_FOLLOWERS}[0] ${PORT} ${100}
+ Buy Cars And Verify @{CAR_FOLLOWERS}[0] ${NUM_ENTRIES}
-Check Contents of Leader Shards
+Check Contents Of Car Leader Shards
[Documentation] Check all content using Leader
- wait until keyword succeeds 30 1 Check Cars ${CURRENT_CAR_LEADER} ${PORT} 100
- wait until keyword succeeds 30 1 Check People ${CURRENT_CAR_LEADER} ${PORT} 100
- wait until keyword succeeds 30 1 Check CarPeople ${CURRENT_CAR_LEADER} ${PORT} 100
+ Check Elements In Shards ${CURRENT_CAR_LEADER} ${NUM_ENTRIES}
-Check Contents of First Follower Shards
+Check Contents Of Car Follower1 Shards
[Documentation] Check all content using first follower
- wait until keyword succeeds 30 1 Check Cars @{CAR_FOLLOWERS}[0] ${PORT} 100
- wait until keyword succeeds 30 1 Check People @{CAR_FOLLOWERS}[0] ${PORT} 100
- wait until keyword succeeds 30 1 Check CarPeople @{CAR_FOLLOWERS}[0] ${PORT} 100
+ Check Elements In Shards @{CAR_FOLLOWERS}[0] ${NUM_ENTRIES}
-Check Contents of Second Follower Shards
+Check Contents Of Car Follower2 Shards
[Documentation] Check all content using second follower
- wait until keyword succeeds 30 1 Check Cars @{CAR_FOLLOWERS}[1] ${PORT} 100
- wait until keyword succeeds 30 1 Check People @{CAR_FOLLOWERS}[1] ${PORT} 100
- wait until keyword succeeds 30 1 Check CarPeople @{CAR_FOLLOWERS}[1] ${PORT} 100
+ Check Elements In Shards @{CAR_FOLLOWERS}[1] ${NUM_ENTRIES}
-Switch Car leader
- [Documentation] Stop the leader to cause a new leader to be elected
- ${OLD_CAR_LEADER}= Set Variable ${CURRENT_CAR_LEADER}
+Get Old Car Leader
+ [Documentation] Find leader in the car shard
+ ${OLD_CAR_LEADER} Get Leader And Verify ${SHARD_CAR_NAME}
Set Suite Variable ${OLD_CAR_LEADER}
- ${NEW_CAR_LEADER} Switch Leader ${SHARD_CAR_NAME} ${CURRENT_CAR_LEADER}
+
+Switch Car Leader
+ [Documentation] Stop the leader to cause a new leader to be elected
+ Stop One Or More Controllers ${OLD_CAR_LEADER}
+ Wait For Controller Down ${STOP_TIMEOUT} ${OLD_CAR_LEADER}
+ ${NEW_CAR_LEADER} Wait Until Keyword Succeeds 30s 2s Get Leader And Verify ${SHARD_CAR_NAME} ${OLD_CAR_LEADER}
Set Suite Variable ${NEW_CAR_LEADER}
-Get New Car Follower
- [Documentation] Find the new leader for the car shard.
+Get New Car Followers
+ [Documentation] Find the new followers for the car shard.
@{CAR_FOLLOWERS} Get All Followers ${SHARD_CAR_NAME}
Set Suite Variable ${CAR_FOLLOWERS}
Log @{CAR_FOLLOWERS}[0]
-Overwrite cars and get cars from New Leader
- [Documentation] Overwrite 100 cars and get added cars from Leader
- ${resp}= AddCar ${NEW_CAR_LEADER} ${PORT} ${100}
- ${resp}= Getcars ${NEW_CAR_LEADER} ${PORT} ${0}
- Should Be Equal As Strings ${resp.status_code} 200
- Should Contain ${resp.content} manufacturer1 cars not added!
-
-Overwrite persons and get persons from New Leader
- [Documentation] Overwrite 100 persons and get persons Note: There should be one person added first to enable rpc
- ${resp} AddPerson ${NEW_CAR_LEADER} ${PORT} ${100}
- ${resp} GetPersons ${NEW_CAR_LEADER} ${PORT} ${0}
- Should Be Equal As Strings ${resp.status_code} 200
- Should Contain ${resp.content} user5 people not added!
-
-RePurchase 100 cars using New Follower1
- [Documentation] RePurchase 100 cars using Follower1
- BuyCar @{CAR_FOLLOWERS}[0] ${PORT} ${100}
-
-Check Contents of New Leader Shards
- [Documentation] Check all content using Leader
- Log ${NEW_CAR_LEADER}
- wait until keyword succeeds 30 1 Check Cars ${NEW_CAR_LEADER} ${PORT} 100
- wait until keyword succeeds 30 1 Check People ${NEW_CAR_LEADER} ${PORT} 100
- wait until keyword succeeds 30 1 Check CarPeople ${NEW_CAR_LEADER} ${PORT} 100
+Check Cars In New Car Leader
+ [Documentation] Check cars in new Leader
+ Wait Until Keyword Succeeds ${START_TIMEOUT} 2s Check Cars ${NEW_CAR_LEADER} ${NUM_ENTRIES}
+
+Check Contents Of New Car Leader Shards
+ [Documentation] Check all content using new Leader
+ Check Elements In Shards ${NEW_CAR_LEADER} ${NUM_ENTRIES}
+
+Check Contents Of New Car Follower1 Shards
+ [Documentation] Check all content using first follower
+ Check Elements In Shards @{CAR_FOLLOWERS}[0] ${NUM_ENTRIES}
-Check Contents of New First Follower Shards
+Readd People From New Car Leader
+ [Documentation] Add 100 persons and get persons from Leader
+ Add People And Verify Without Init ${NEW_CAR_LEADER} ${NUM_ENTRIES}
+
+Repurchase 100 Cars Using New Car Follower1
+ [Documentation] Repurchase 100 cars using Follower1
+ Buy Cars And Verify @{CAR_FOLLOWERS}[0] ${NUM_ENTRIES}
+
+Check Contents Of New Car Leader Shards After Repurchase
+ [Documentation] Check all content using new Leader
+ Check Elements In Shards ${NEW_CAR_LEADER} ${NUM_ENTRIES}
+
+Check Contents Of New Car First Follower Shards After Repurchase
[Documentation] Check all content using first follower
- wait until keyword succeeds 30 1 Check Cars @{CAR_FOLLOWERS}[0] ${PORT} 100
- wait until keyword succeeds 30 1 Check People @{CAR_FOLLOWERS}[0] ${PORT} 100
- wait until keyword succeeds 30 1 Check CarPeople @{CAR_FOLLOWERS}[0] ${PORT} 100
+ Check Elements In Shards @{CAR_FOLLOWERS}[0] ${NUM_ENTRIES}
-Start Leader
+Start Old Car Leader
[Documentation] Start Leader controller
Start One Or More Controllers ${OLD_CAR_LEADER}
+ Wait For Controller Sync ${START_TIMEOUT} ${OLD_CAR_LEADER}
+
+Check Cars In Old Car Leader
+ [Documentation] Check cars in old Leader
+ Wait Until Keyword Succeeds ${START_TIMEOUT} 2s Check Cars ${OLD_CAR_LEADER} ${NUM_ENTRIES}
+
+Check Contents Of Old Leader Shards
+ [Documentation] Check all content using old Leader
+ Check Elements In Shards ${OLD_CAR_LEADER} ${NUM_ENTRIES}
+
+Readd People From Old Leader
+ [Documentation] Add 100 persons and get persons from Leader
+ Wait Until Keyword Succeeds 30 2s Add People And Verify Without Init ${OLD_CAR_LEADER} ${NUM_ENTRIES}
+
+Repurchase 100 Cars Using Old Leader
+ [Documentation] Repurchase 100 cars using Follower1
+ Buy Cars And Verify ${OLD_CAR_LEADER} ${NUM_ENTRIES}
+
+Check Contents Of Old Leader Shards After Repurchase
+ [Documentation] Check all content using Leader
+ Check Elements In Shards ${OLD_CAR_LEADER} ${NUM_ENTRIES}
+
Code Review / integration / test.git / commitdiff