From 52023e4743295b8ffba4db4f1ecbd1234b76e469 Mon Sep 17 00:00:00 2001 From: Luis Gomez Date: Wed, 7 Oct 2015 17:52:13 -0700 Subject: [PATCH] Refactor cluster car-people test to reduce test time in case of failure Reorganize and clean up cluster keywords. Introduce START_TIMEOUT and STOP_TIMEOUT variables for instance restart. Use cars to determine whether application is up. Remove unnecessary waits. Add test case documentation. Change test cases name to standard. Change-Id: I819cb86c604338b7452f4378ecde82d24cde6c16 Signed-off-by: Luis Gomez --- csit/libraries/CarsAndPeople.robot | 241 ++++++++++++++++ csit/libraries/CarsAndPeopleCRUD.robot | 73 ----- csit/libraries/ClusterKeywords.robot | 269 +++++------------- .../001_start_cluster.robot | 4 + .../010_crud_on_leader.robot | 58 ++-- .../020_crud_on_any_follower.robot | 45 ++- .../030_car_failover_crud_on_new_leader.robot | 56 ++-- ...0_people_failover_crud_on_new_leader.robot | 54 ++-- .../050_car_persistence_recovery.robot | 68 +++-- .../140_recovery_restart_follower.robot | 49 ++-- .../023_routed_rpc_crud_test.robot | 168 +++++------ 11 files changed, 591 insertions(+), 494 deletions(-) create mode 100644 csit/libraries/CarsAndPeople.robot delete mode 100644 csit/libraries/CarsAndPeopleCRUD.robot diff --git a/csit/libraries/CarsAndPeople.robot b/csit/libraries/CarsAndPeople.robot new file mode 100644 index 0000000000..a95d0afc68 --- /dev/null +++ b/csit/libraries/CarsAndPeople.robot @@ -0,0 +1,241 @@ +*** Settings *** +Library Collections +Library CrudLibrary.py +Library SettingsLibrary.py +Resource DatastoreCRUD.robot + +*** Keywords *** +Add Cars And Verify + [Arguments] ${controller_ip} ${num_cars} ${timeout}=3s + [Documentation] Initializes shard and then adds the specified number of cars and performs a GET as a check. + ${resp} InitCar ${controller_ip} ${PORT} + Should Be Equal As Strings ${resp.status_code} 204 + ${resp} AddCar ${controller_ip} ${RESTCONFPORT} ${num_cars} 204 + Should Be Equal As Strings ${resp.status_code} 204 + Wait Until Keyword Succeeds ${timeout} 1s Get Cars And Verify ${controller_ip} ${num_cars} + +Add Cars And Verify Without Init + [Arguments] ${controller_ip} ${num_cars} ${timeout}=3s + [Documentation] Adds cars to an initialized cars shard then performs a GET as a check. + Comment First car add may return 409, but subsequent should be 204 + ${resp} AddCar ${controller_ip} ${RESTCONFPORT} ${num_cars} 204 409 + Should Be Equal As Strings ${resp.status_code} 204 + Wait Until Keyword Succeeds ${timeout} 1s Get Cars And Verify ${controller_ip} ${num_cars} + +Get Cars And Verify + [Arguments] ${controller_ip} ${num_cars} + [Documentation] Gets cars and verifies that the manufacturer is correct. + # TODO: Future enhanement: verify all fields. + ${resp} Getcars ${controller_ip} ${RESTCONFPORT} ${0} + Should Be Equal As Strings ${resp.status_code} 200 + : FOR ${i} IN RANGE 1 ${num_cars}+1 + \ Should Contain ${resp.content} manufacturer${i} + +Add People And Verify + [Arguments] ${controller_ip} ${num_people} ${timeout}=3s + [Documentation] Note: The first AddPerson call passed with 0 posts directly to the data store to get + ... the people container created so the subsequent AddPerson RPC calls that put to the + ... person list will succeed. + ${resp} AddPerson ${controller_ip} ${RESTCONFPORT} ${0} 204 + Should Be Equal As Strings ${resp.status_code} 204 + Wait Until Keyword Succeeds ${timeout} 1s Get One Person And Verify ${controller_ip} ${0} + ${resp} AddPerson ${controller_ip} ${RESTCONFPORT} ${num_people} 200 + Wait Until Keyword Succeeds ${timeout} 1s Get People And Verify ${controller_ip} ${num_people} + +Add People And Verify Without Init + [Arguments] ${controller_ip} ${num_people} ${timeout}=3s + [Documentation] Adds people to an initialized people shard then performs a GET as a check. + ${resp} AddPerson ${controller_ip} ${RESTCONFPORT} ${num_people} 200 + Should Be Equal As Strings ${resp.status_code} 200 + Wait Until Keyword Succeeds ${timeout} 1s Get People And Verify ${controller_ip} ${num_people} + +Get One Person And Verify + [Arguments] ${controller_ip} ${number} + [Documentation] Gets a person and verifies that the user ID is correct. + # TODO: Future enhanement: verify all fields. + ${resp} GetPersons ${controller_ip} ${RESTCONFPORT} ${0} + Should Be Equal As Strings ${resp.status_code} 200 + Should Contain ${resp.content} user${number} + +Get People And Verify + [Arguments] ${controller_ip} ${num_people} + [Documentation] Gets multiple people and verifies that the user IDs are correct. + # TODO: Future enhanement: verify all fields. + ${resp} GetPersons ${controller_ip} ${RESTCONFPORT} ${0} + Should Be Equal As Strings ${resp.status_code} 200 + : FOR ${i} IN RANGE 1 ${num_people}+1 + \ Should Contain ${resp.content} user${i} + +Add Car Person And Verify + [Arguments] ${controller_ip} ${timeout}=3s + [Documentation] Add a car-person via the data store and get the car-person from Leader. + ... Note: This is done to get the car-people container created so subsequent + ... BuyCar RPC puts to the car-person list will succeed. + AddCarPerson ${controller_ip} ${RESTCONFPORT} ${0} + Wait Until Keyword Succeeds ${timeout} 1s Get One Car-Person Mapping And Verify ${controller_ip} ${0} + +Get One Car-Person Mapping And Verify + [Arguments] ${controller_ip} ${number} + [Documentation] Gets a car person mapping and verifies that the user ID is correct. + ${resp} GetCarPersonMappings ${controller_ip} ${RESTCONFPORT} ${0} + Should Be Equal As Strings ${resp.status_code} 200 + Should Contain ${resp.content} user${number} + +Get Car-Person Mappings And Verify + [Arguments] ${controller_ip} ${num_entries} + ${resp} GetCarPersonMappings ${controller_ip} ${RESTCONFPORT} ${0} + Should Be Equal As Strings ${resp.status_code} 200 + : FOR ${i} IN RANGE 1 ${num_entries}+1 + \ Should Contain ${resp.content} user${i} + +Buy Cars And Verify + [Arguments] ${controller_ip} ${num_entries} ${start}=${0} ${timeout}=3s + BuyCar ${controller_ip} ${RESTCONFPORT} ${num_entries} ${start} + ${total_entries} Evaluate ${start}+${num_entries} + Wait Until Keyword Succeeds ${timeout} 1s Get Car-Person Mappings And Verify ${controller_ip} ${total_entries} + +Check Cars Deleted + [Arguments] ${controller_ip} + ${resp} Getcars ${controller_ip} ${RESTCONFPORT} ${0} + Should Be Equal As Strings ${resp.status_code} 404 + +Delete All Cars And Verify + [Arguments] ${controller_ip} ${timeout}=3s + DeleteAllCars ${controller_ip} ${RESTCONFPORT} ${0} + Wait Until Keyword Succeeds ${timeout} 1s Check Cars Deleted ${controller_ip} + +Check People Deleted + [Arguments] ${controller_ip} + ${resp} GetPersons ${controller_ip} ${RESTCONFPORT} ${0} + Should Be Equal As Strings ${resp.status_code} 404 + +Delete All People And Verify + [Arguments] ${controller_ip} ${timeout}=3s + DeleteAllPersons ${controller_ip} ${RESTCONFPORT} ${0} + Wait Until Keyword Succeeds ${timeout} 1s Check People Deleted ${controller_ip} + +Check Cars-Persons Deleted + [Arguments] ${controller_ip} + ${resp} GetCarPersonMappings ${controller_ip} ${RESTCONFPORT} ${0} + Should Be Equal As Strings ${resp.status_code} 404 + +Delete All Cars-Persons And Verify + [Arguments] ${controller_ip} ${timeout}=3s + DeleteAllCarsPersons ${controller_ip} ${RESTCONFPORT} ${0} + Wait Until Keyword Succeeds ${timeout} 1s Check Cars-Persons Deleted ${controller_ip} + +Delete All Entries From Shards + [Documentation] Delete All Shards. + [Arguments] @{controllers} + : FOR ${ip} IN @{controllers} + \ Delete All Cars And Verify ${ip} + : FOR ${ip} IN @{controllers} + \ Delete All People And Verify ${ip} + : FOR ${ip} IN @{controllers} + \ Delete All Cars-Persons And Verify ${ip} + +Check Cars + [Arguments] ${controller_ip} ${nth_car} + [Documentation] Verifies that the first through nth car is present. + ${resp} Getcars ${controller_ip} ${RESTCONFPORT} ${0} + Should Be Equal As Strings ${resp.status_code} 200 + : FOR ${INDEX} IN RANGE 1 ${nth_car} + \ ${counter}= Convert to String ${INDEX} + \ Log manufacturer${counter} + \ Should Contain ${resp.content} manufacturer${counter} + +Check People + [Arguments] ${controller_ip} ${nth_person} + [Documentation] Verifies that the first through nth person is present. + ${resp} GetPersons ${controller_ip} ${RESTCONFPORT} ${0} + Should Be Equal As Strings ${resp.status_code} 200 + : FOR ${INDEX} IN RANGE 1 ${nth_person} + \ ${counter}= Convert to String ${INDEX} + \ Log user${counter} + \ Should Contain ${resp.content} user${counter} + +Check CarPeople + [Arguments] ${controller_ip} ${nth_carperson} + [Documentation] Verifies that the first through nth car-person is present. + ${resp} GetCarPersonMappings ${controller_ip} ${RESTCONFPORT} ${0} + Should Be Equal As Strings ${resp.status_code} 200 + : FOR ${INDEX} IN RANGE 1 ${nth_carperson} + \ ${counter}= Convert to String ${INDEX} + \ Log user${counter} + \ Should Contain ${resp.content} user${counter} + +Check Elements In Shards + [Arguments] ${controller_ip} ${nth} + [Documentation] Check all shards for nth elements + wait until keyword succeeds 3 1 Check Cars ${controller_ip} ${nth} + wait until keyword succeeds 3 1 Check People ${controller_ip} ${nth} + wait until keyword succeeds 3 1 Check CarPeople ${controller_ip} ${nth} + +Initialize Cars + [Arguments] ${controller_ip} ${field bases} + [Documentation] Initializes the cars shard by creating a 0th car with POST then deleting it. + ... Field bases are a dictionary of datastore record field values onto which is appended + ... an incremental value to uniquely identify the record from which it came. + ... Typically, you will use the Create Dictionary keyword on arguments which look like this: + ... id=${EMPTY} category=coupe model=model manufacturer=mfg year=2 + ${node}= Set Variable ${EMPTY} + ${prefix}= Set Variable {"car:cars":{"car-entry":[{ + ${postfix}= Set Variable }]}} + Create Records ${controller_ip} ${node} ${0} ${0} ${prefix} ${field bases} + ... ${postfix} + ${node}= Set Variable car:cars/car-entry + Delete Records ${controller_ip} ${node} ${0} ${0} + +Create Cars + [Arguments] ${controller_ip} ${first} ${last} ${field bases} + [Documentation] Creates cars with record IDs of specified range using POST. + ... If first and last are equal, only one record is updated. + ... Field bases are a dictionary of datastore record field values onto which is appended + ... an incremental value to uniquely identify the record from which it came. + ... Typically, you will use the Create Dictionary keyword on an argument which looks like this: + ... id=${EMPTY} category=coupe model=model manufacturer=mfg year=2 + ${node}= Set Variable car:cars + ${prefix}= Set Variable {"car-entry":[{ + ${postfix}= Set Variable }]} + Create Records ${controller_ip} ${node} ${first} ${last} ${prefix} ${field bases} + ... ${postfix} + +Update Cars + [Arguments] ${controller_ip} ${first} ${last} ${field bases} + [Documentation] Updates cars with record IDs of the specified using PUT. + ... If first and last are equal, only one record is updated. + ... Field bases are a dictionary of datastore record field values onto which is appended + ... an incremental value to uniquely identify the record from which it came. + ... Typically, you will use the Create Dictionary keyword on arguments which look like this: + ... id=${EMPTY} category=coupe model=model manufacturer=mfg year=2 + ${node}= Set Variable car:cars/car-entry + ${prefix}= Set Variable {"car-entry":[{ + ${postfix}= Set Variable }]} + Update Records ${controller_ip} ${node} ${first} ${last} ${prefix} ${field bases} + ... ${postfix} + +Read All Cars + [Arguments] ${controller_ip} + [Documentation] Returns all records from the cars shard in JSON format. + ${node}= Set Variable car:cars + ${result}= Read Records ${controller_ip} ${node} + [Return] ${result} + +Read One Car + [Arguments] ${controller_ip} ${id} + [Documentation] Returns the specified record from the cars shard in JSON format. + ${node}= Set Variable car:cars/car-entry/${id} + ${result}= Read Records ${controller_ip} ${node} + [Return] ${result} + +Remove All Cars + [Arguments] ${controller_ip} + [Documentation] Deletes all records from the cars shard. + ${node}= Set Variable car:cars + Delete All Records ${controller_ip} ${node} + +Remove Cars + [Arguments] ${controller_ip} ${first} ${last} + [Documentation] Deletes the specified range of records from the cars shard. + ${node}= Set Variable car:cars/car-entry + Delete Records ${controller_ip} ${node} ${first} ${last} diff --git a/csit/libraries/CarsAndPeopleCRUD.robot b/csit/libraries/CarsAndPeopleCRUD.robot deleted file mode 100644 index dddd3df824..0000000000 --- a/csit/libraries/CarsAndPeopleCRUD.robot +++ /dev/null @@ -1,73 +0,0 @@ -*** Settings *** -Library Collections -Resource DatastoreCRUD.robot - -*** Keywords *** -Initialize Cars - [Arguments] ${controller_ip} ${field bases} - [Documentation] Initializes the cars shard by creating a 0th car with POST then deleting it. - ... Field bases are a dictionary of datastore record field values onto which is appended - ... an incremental value to uniquely identify the record from which it came. - ... Typically, you will use the Create Dictionary keyword on arguments which look like this: - ... id=${EMPTY} category=coupe model=model manufacturer=mfg year=2 - ${node}= Set Variable ${EMPTY} - ${prefix}= Set Variable {"car:cars":{"car-entry":[{ - ${postfix}= Set Variable }]}} - Create Records ${controller_ip} ${node} ${0} ${0} ${prefix} ${field bases} - ... ${postfix} - ${node}= Set Variable car:cars/car-entry - Delete Records ${controller_ip} ${node} ${0} ${0} - -Create Cars - [Arguments] ${controller_ip} ${first} ${last} ${field bases} - [Documentation] Creates cars with record IDs of specified range using POST. - ... If first and last are equal, only one record is updated. - ... Field bases are a dictionary of datastore record field values onto which is appended - ... an incremental value to uniquely identify the record from which it came. - ... Typically, you will use the Create Dictionary keyword on an argument which looks like this: - ... id=${EMPTY} category=coupe model=model manufacturer=mfg year=2 - ${node}= Set Variable car:cars - ${prefix}= Set Variable {"car-entry":[{ - ${postfix}= Set Variable }]} - Create Records ${controller_ip} ${node} ${first} ${last} ${prefix} ${field bases} - ... ${postfix} - -Update Cars - [Arguments] ${controller_ip} ${first} ${last} ${field bases} - [Documentation] Updates cars with record IDs of the specified using PUT. - ... If first and last are equal, only one record is updated. - ... Field bases are a dictionary of datastore record field values onto which is appended - ... an incremental value to uniquely identify the record from which it came. - ... Typically, you will use the Create Dictionary keyword on arguments which look like this: - ... id=${EMPTY} category=coupe model=model manufacturer=mfg year=2 - ${node}= Set Variable car:cars/car-entry - ${prefix}= Set Variable {"car-entry":[{ - ${postfix}= Set Variable }]} - Update Records ${controller_ip} ${node} ${first} ${last} ${prefix} ${field bases} - ... ${postfix} - -Read All Cars - [Arguments] ${controller_ip} - [Documentation] Returns all records from the cars shard in JSON format. - ${node}= Set Variable car:cars - ${result}= Read Records ${controller_ip} ${node} - [Return] ${result} - -Read One Car - [Arguments] ${controller_ip} ${id} - [Documentation] Returns the specified record from the cars shard in JSON format. - ${node}= Set Variable car:cars/car-entry/${id} - ${result}= Read Records ${controller_ip} ${node} - [Return] ${result} - -Delete All Cars - [Arguments] ${controller_ip} - [Documentation] Deletes all records from the cars shard. - ${node}= Set Variable car:cars - Delete All Records ${controller_ip} ${node} - -Delete Cars - [Arguments] ${controller_ip} ${first} ${last} - [Documentation] Deletes the specified range of records from the cars shard. - ${node}= Set Variable car:cars/car-entry - Delete Records ${controller_ip} ${node} ${first} ${last} diff --git a/csit/libraries/ClusterKeywords.robot b/csit/libraries/ClusterKeywords.robot index d80ebe6f6f..270e23888f 100644 --- a/csit/libraries/ClusterKeywords.robot +++ b/csit/libraries/ClusterKeywords.robot @@ -1,8 +1,9 @@ *** Settings *** -Resource Utils.robot Library RequestsLibrary Library Collections +Library UtilLibrary.py Library ClusterStateLibrary.py +Resource Utils.robot *** Variables *** ${smc_node} /org.opendaylight.controller:Category=ShardManager,name=shard-manager-config,type=DistributedConfigDatastore @@ -22,167 +23,35 @@ Get Leader And Verify [Documentation] Returns the IP addr or hostname of the leader of the specified shard. ... Controllers are specifed in the pybot command line. ${searchlist} Get Controller List ${old_leader} - ${leader} GetLeader ${shard_name} ${3} ${3} ${1} ${PORT} + ${leader} GetLeader ${shard_name} ${3} ${3} ${1} ${RESTCONFPORT} ... @{searchlist} Should Not Be Equal As Strings ${leader} None Run Keyword If '${old_leader}'!='${EMPTY}' Should Not Be Equal ${old_leader} ${leader} [Return] ${leader} -Wait For Leader To Be Found +Expect No Leader + [Documentation] No leader is elected in the car shard [Arguments] ${shard_name} - [Documentation] Waits until the leader of the specified shard is found. - ${leader} Wait Until Keyword Succeeds 12s 2s Get Leader And Verify ${shard_name} - Log ${leader} - [Return] ${leader} - -Switch Leader - [Arguments] ${shard_name} ${current_leader} - [Documentation] Forces a change of leadership by shutting down the current leader. - Stop One Or More Controllers ${current_leader} - ${new_leader} Wait Until Keyword Succeeds 60s 2s Get Leader And Verify ${shard_name} ${current_leader} - # TODO: Future enhanement: make sure the other controller is a follower and not a master or candidate. - Log ${new_leader} - [Return] ${new_leader} + ${leader} GetLeader ${shard_name} ${3} ${1} ${1} ${RESTCONFPORT} + ... ${CURRENT_CAR_LEADER} + Should Be Equal As Strings ${leader} None Get All Followers [Arguments] ${shard_name} ${exclude_controller}=${EMPTY} [Documentation] Returns the IP addresses or hostnames of all followers of the specified shard. ${searchlist} Get Controller List ${exclude_controller} - ${followers} GetFollowers ${shard_name} ${3} ${3} ${1} ${PORT} + ${followers} GetFollowers ${shard_name} ${3} ${3} ${1} ${RESTCONFPORT} ... @{searchlist} Log ${followers} Should Not Be Empty ${followers} [Return] ${followers} -Add Cars And Verify - [Arguments] ${controller_ip} ${num_cars} ${timeout}=12s - [Documentation] Initializes shard and then adds the specified number of cars and performs a GET as a check. - ${resp} InitCar ${controller_ip} ${PORT} - Should Be Equal As Strings ${resp.status_code} 204 - ${resp} AddCar ${controller_ip} ${RESTCONFPORT} ${num_cars} 204 - Should Be Equal As Strings ${resp.status_code} 204 - Wait Until Keyword Succeeds ${timeout} 2s Get Cars And Verify ${controller_ip} ${num_cars} - -Add Cars And Verify Without Init - [Arguments] ${controller_ip} ${num_cars} ${timeout}=12s - [Documentation] Adds cars to an initialized cars shard then performs a GET as a check. - Comment First car add may return 409, but subsequent should be 204 - ${resp} AddCar ${controller_ip} ${RESTCONFPORT} ${num_cars} 204 409 - Should Be Equal As Strings ${resp.status_code} 204 - Wait Until Keyword Succeeds ${timeout} 2s Get Cars And Verify ${controller_ip} ${num_cars} - -Get Cars And Verify - [Arguments] ${controller_ip} ${num_cars} - [Documentation] Gets cars and verifies that the manufacturer is correct. - # TODO: Future enhanement: verify all fields. - ${resp} Getcars ${controller_ip} ${RESTCONFPORT} ${0} - Should Be Equal As Strings ${resp.status_code} 200 - : FOR ${i} IN RANGE 1 ${num_cars}+1 - \ Should Contain ${resp.content} manufacturer${i} - -Add People And Verify - [Arguments] ${controller_ip} ${num_people} - [Documentation] Note: The first AddPerson call passed with 0 posts directly to the data store to get - ... the people container created so the subsequent AddPerson RPC calls that put to the - ... person list will succeed. - ${resp} AddPerson ${controller_ip} ${RESTCONFPORT} ${0} 204 - Should Be Equal As Strings ${resp.status_code} 204 - Wait Until Keyword Succeeds 12s 2s Get One Person And Verify ${controller_ip} ${0} - ${resp} AddPerson ${controller_ip} ${RESTCONFPORT} ${num_people} 200 - Wait Until Keyword Succeeds 12s 2s Get People And Verify ${controller_ip} ${num_people} - -Get One Person And Verify - [Arguments] ${controller_ip} ${number} - [Documentation] Gets a person and verifies that the user ID is correct. - # TODO: Future enhanement: verify all fields. - ${resp} GetPersons ${controller_ip} ${RESTCONFPORT} ${0} - Should Be Equal As Strings ${resp.status_code} 200 - Should Contain ${resp.content} user${number} - -Get People And Verify - [Arguments] ${controller_ip} ${num_people} - [Documentation] Gets multiple people and verifies that the user IDs are correct. - # TODO: Future enhanement: verify all fields. - ${resp} GetPersons ${controller_ip} ${RESTCONFPORT} ${0} - Should Be Equal As Strings ${resp.status_code} 200 - : FOR ${i} IN RANGE 1 ${num_people}+1 - \ Should Contain ${resp.content} user${i} - -Add Car Person And Verify - [Arguments] ${controller_ip} - [Documentation] Add a car-person via the data store and get the car-person from Leader. - ... Note: This is done to get the car-people container created so subsequent - ... BuyCar RPC puts to the car-person list will succeed. - AddCarPerson ${controller_ip} ${RESTCONFPORT} ${0} - Wait Until Keyword Succeeds 60s 2s Get One Car-Person Mapping And Verify ${controller_ip} ${0} - -Get One Car-Person Mapping And Verify - [Arguments] ${controller_ip} ${number} - [Documentation] Gets a car person mapping and verifies that the user ID is correct. - ${resp} GetCarPersonMappings ${controller_ip} ${RESTCONFPORT} ${0} - Should Be Equal As Strings ${resp.status_code} 200 - Should Contain ${resp.content} user${number} - -Get Car-Person Mappings And Verify - [Arguments] ${controller_ip} ${num_entries} - ${resp} GetCarPersonMappings ${controller_ip} ${RESTCONFPORT} ${0} - Should Be Equal As Strings ${resp.status_code} 200 - : FOR ${i} IN RANGE 1 ${num_entries}+1 - \ Should Contain ${resp.content} user${i} - -Buy Cars And Verify - [Arguments] ${controller_ip} ${num_entries} ${start}=${0} - Wait Until Keyword Succeeds 60s 2s BuyCar ${controller_ip} ${RESTCONFPORT} ${num_entries} - ... ${start} - -Check Cars Deleted - [Arguments] ${controller_ip} - ${resp} Getcars ${controller_ip} ${RESTCONFPORT} ${0} - Should Be Equal As Strings ${resp.status_code} 404 - -Delete All Cars And Verify - [Arguments] ${controller_ip} - DeleteAllCars ${controller_ip} ${RESTCONFPORT} ${0} - Wait Until Keyword Succeeds 60s 2s Check Cars Deleted ${controller_ip} - -Check People Deleted - [Arguments] ${controller_ip} - ${resp} GetPersons ${controller_ip} ${RESTCONFPORT} ${0} - Should Be Equal As Strings ${resp.status_code} 404 - -Delete All People And Verify - [Arguments] ${controller_ip} - DeleteAllPersons ${controller_ip} ${RESTCONFPORT} ${0} - Wait Until Keyword Succeeds 60s 2s Check People Deleted ${controller_ip} - -Check Cars-Persons Deleted - [Arguments] ${controller_ip} - ${resp} GetCarPersonMappings ${controller_ip} ${RESTCONFPORT} ${0} - Should Be Equal As Strings ${resp.status_code} 404 - -Delete All Cars-Persons And Verify - [Arguments] ${controller_ip} - DeleteAllCarsPersons ${controller_ip} ${RESTCONFPORT} ${0} - Wait Until Keyword Succeeds 60s 2s Check Cars-Persons Deleted ${controller_ip} - Stop One Or More Controllers [Arguments] @{controllers} [Documentation] Give this keyword a scalar or list of controllers to be stopped. ${cmd} = Set Variable ${KARAF_HOME}/bin/stop : FOR ${ip} IN @{controllers} \ Run Command On Remote System ${ip} ${cmd} - : FOR ${ip} IN @{controllers} - \ Wait Until Keyword Succeeds ${CONTROLLER_STOP_TIMEOUT} s 3 s Controller Down Check ${ip} - -Start One Or More Controllers - [Arguments] @{controllers} - [Documentation] Give this keyword a scalar or list of controllers to be started. - ${cmd} = Set Variable ${KARAF_HOME}/bin/start - : FOR ${ip} IN @{controllers} - \ Run Command On Remote System ${ip} ${cmd} - # TODO: This should throw an error if controller never comes up. - : FOR ${ip} IN @{controllers} - \ UtilLibrary.Wait For Controller Up ${ip} ${RESTCONFPORT} Kill One Or More Controllers [Arguments] @{controllers} @@ -191,8 +60,18 @@ Kill One Or More Controllers log ${cmd} : FOR ${ip} IN @{controllers} \ Run Command On Remote System ${ip} ${cmd} + +Wait For Cluster Down + [Arguments] ${timeout} @{controllers} + [Documentation] Waits for one or more clustered controllers to be down. : FOR ${ip} IN @{controllers} - \ Wait Until Keyword Succeeds 12 s 3 s Controller Down Check ${ip} + \ ${status}= Run Keyword And Return Status Wait For Controller Down ${timeout} ${ip} + \ Exit For Loop If '${status}' == 'FAIL' + +Wait For Controller Down + [Arguments] ${timeout} ${ip} + [Documentation] Waits for one controllers to be down. + Wait Until Keyword Succeeds ${timeout} 2s Controller Down Check ${ip} Controller Down Check [Arguments] ${ip} @@ -202,6 +81,52 @@ Controller Down Check Log Number of controller instances running: ${response} Should Start With ${response} 0 Controller process found or there may be extra instances of karaf running on the host machine. +Start One Or More Controllers + [Arguments] @{controllers} + [Documentation] Give this keyword a scalar or list of controllers to be started. + ${cmd} = Set Variable ${KARAF_HOME}/bin/start + : FOR ${ip} IN @{controllers} + \ Run Command On Remote System ${ip} ${cmd} + +Wait For Cluster Sync + [Arguments] ${timeout} @{controllers} + [Documentation] Waits for one or more clustered controllers to report Sync Status as true. + : FOR ${ip} IN @{controllers} + \ ${status}= Run Keyword And Return Status Wait For Controller Sync ${timeout} ${ip} + \ Exit For Loop If '${status}' == 'FAIL' + +Wait For Controller Sync + [Arguments] ${timeout} ${ip} + [Documentation] Waits for one controllers to report Sync Status as true. + Wait Until Keyword Succeeds ${timeout} 2s Controller Sync Status Should Be True ${ip} + +Controller Sync Status Should Be True + [Arguments] ${ip} + [Documentation] Checks if Sync Status is true. + ${SyncStatus}= Get Controller Sync Status ${ip} + Should Be Equal ${SyncStatus} ${True} + +Controller Sync Status Should Be False + [Arguments] ${ip} + [Documentation] Checks if Sync Status is false. + ${SyncStatus}= Get Controller Sync Status ${ip} + Should Be Equal ${SyncStatus} ${False} + +Get Controller Sync Status + [Arguments] ${controller_ip} + [Documentation] Return Sync Status. + ${api} Set Variable /jolokia/read + Create_Session session http://${controller_ip}:${RESTCONFPORT}${api} headers=${HEADERS} auth=${AUTH} + ${resp}= RequestsLibrary.Get session ${smc_node} + Log ${resp.json()} + Log ${resp.content} + ${json}= Set Variable ${resp.json()} + ${value}= Get From Dictionary ${json} value + Log value: ${value} + ${SyncStatus}= Get From Dictionary ${value} SyncStatus + Log SyncSatus: ${SyncStatus} + [Return] ${SyncStatus} + Clean One Or More Journals [Arguments] @{controllers} [Documentation] Give this keyword a scalar or list of controllers on which to clean journals. @@ -239,36 +164,6 @@ Show Cluster Configuation Files : FOR ${ip} IN @{controllers} \ Run Command On Remote System ${ip} ${cmd} -Check Cars - [Arguments] ${selected controller} ${PORT} ${nth car} - [Documentation] Verifies that the first through nth car is present. - ${resp} Getcars ${selected controller} ${PORT} ${0} - Should Be Equal As Strings ${resp.status_code} 200 - : FOR ${INDEX} IN RANGE 1 ${nth car} - \ ${counter}= Convert to String ${INDEX} - \ Log manufacturer${counter} - \ Should Contain ${resp.content} manufacturer${counter} - -Check People - [Arguments] ${selected controller} ${PORT} ${nth person} - [Documentation] Verifies that the first through nth person is present. - ${resp} GetPersons ${selected controller} ${PORT} ${0} - Should Be Equal As Strings ${resp.status_code} 200 - : FOR ${INDEX} IN RANGE 1 ${nth person} - \ ${counter}= Convert to String ${INDEX} - \ Log user${counter} - \ Should Contain ${resp.content} user${counter} - -Check CarPeople - [Arguments] ${selected controller} ${PORT} ${nth carperson} - [Documentation] Verifies that the first through nth car-person is present. - ${resp} GetCarPersonMappings ${selected controller} ${PORT} ${0} - Should Be Equal As Strings ${resp.status_code} 200 - : FOR ${INDEX} IN RANGE 1 ${nth carperson} - \ ${counter}= Convert to String ${INDEX} - \ Log user${counter} - \ Should Contain ${resp.content} user${counter} - Isolate a Controller From Cluster [Arguments] ${isolated controller} @{controllers} [Documentation] Use IPTables to isolate one controller from the cluster. @@ -329,35 +224,3 @@ Flush IPTables Should Contain ${return string} Flushing chain `FORWARD' Should Contain ${return string} Flushing chain `OUTPUT' -Wait for Cluster Sync - [Arguments] ${timeout} @{controllers} - [Documentation] Waits for one or more clustered controllers to report Sync Status as true. - : FOR ${ip} IN @{controllers} - \ ${resp}= Wait Until Keyword Succeeds ${timeout} 2s Controller Sync Status Should Be True ${ip} - -Controller Sync Status Should Be True - [Arguments] ${ip} - [Documentation] Checks if Sync Status is true. - ${SyncStatus}= Get Controller Sync Status ${ip} - Should Be Equal ${SyncStatus} ${True} - -Controller Sync Status Should Be False - [Arguments] ${ip} - [Documentation] Checks if Sync Status is false. - ${SyncStatus}= Get Controller Sync Status ${ip} - Should Be Equal ${SyncStatus} ${False} - -Get Controller Sync Status - [Arguments] ${controller_ip} - [Documentation] Return Sync Status. - ${api} Set Variable /jolokia/read - Create_Session session http://${controller_ip}:${RESTCONFPORT}${api} headers=${HEADERS} auth=${AUTH} - ${resp}= RequestsLibrary.Get session ${smc_node} - Log ${resp.json()} - Log ${resp.content} - ${json}= Set Variable ${resp.json()} - ${value}= Get From Dictionary ${json} value - Log value: ${value} - ${SyncStatus}= Get From Dictionary ${value} SyncStatus - Log SyncSatus: ${SyncStatus} - [Return] ${SyncStatus} diff --git a/csit/suites/controller/Clustering_Datastore/001_start_cluster.robot b/csit/suites/controller/Clustering_Datastore/001_start_cluster.robot index e9cbe6237c..a4dae47c69 100644 --- a/csit/suites/controller/Clustering_Datastore/001_start_cluster.robot +++ b/csit/suites/controller/Clustering_Datastore/001_start_cluster.robot @@ -6,11 +6,14 @@ Resource ../../../libraries/ClusterKeywords.robot *** Variables *** @{controllers} ${CONTROLLER} ${CONTROLLER1} ${CONTROLLER2} ${KARAF_HOME} ${WORKSPACE}/${BUNDLEFOLDER} +${START_TIMEOUT} 300s +${STOP_TIMEOUT} 180s *** Test Cases *** Stop All Controllers [Documentation] Stop all the controllers in the cluster. Stop One Or More Controllers @{controllers} + Wait For Cluster Down ${STOP_TIMEOUT} @{controllers} Clean All Journals [Documentation] Clean the journals of all the controllers in the cluster @@ -19,3 +22,4 @@ Clean All Journals Start All Controllers [Documentation] Start all the controllers in the cluster Start One Or More Controllers @{controllers} + Wait For Cluster Sync ${START_TIMEOUT} @{controllers} diff --git a/csit/suites/controller/Clustering_Datastore/010_crud_on_leader.robot b/csit/suites/controller/Clustering_Datastore/010_crud_on_leader.robot index e14e07af88..32fc8672be 100644 --- a/csit/suites/controller/Clustering_Datastore/010_crud_on_leader.robot +++ b/csit/suites/controller/Clustering_Datastore/010_crud_on_leader.robot @@ -1,14 +1,8 @@ *** Settings *** Documentation This test finds the leader for shards in a 3-Node cluster and executes CRUD operations on them Default Tags 3-node-cluster -Library Collections -Library RequestsLibrary -Library ../../../libraries/Common.py -Library ../../../libraries/CrudLibrary.py -Library ../../../libraries/SettingsLibrary.py -Library ../../../libraries/UtilLibrary.py -Library ../../../libraries/ClusterStateLibrary.py Resource ../../../libraries/ClusterKeywords.robot +Resource ../../../libraries/CarsAndPeople.robot Variables ../../../variables/Variables.py *** Variables *** @@ -19,51 +13,55 @@ ${NUM_ENTRIES} ${30} *** Test Cases *** Get Car Leader And Followers - ${CURRENT_CAR_LEADER} Wait For Leader To Be Found ${SHARD_CAR_NAME} + [Documentation] Find leader and followers in the car shard + ${CURRENT_CAR_LEADER} Get Leader And Verify ${SHARD_CAR_NAME} Set Suite Variable ${CURRENT_CAR_LEADER} ${CAR_FOLLOWERS} Get All Followers ${SHARD_CAR_NAME} Set Suite Variable ${CAR_FOLLOWERS} -Add cars and get cars from Leader +Add Cars And Get Cars From Leader [Documentation] Add some cars and get added cars from Leader Add Cars And Verify ${CURRENT_CAR_LEADER} ${NUM_ENTRIES} -Get added cars from Follower1 +Get Added Cars From Follower1 [Documentation] Get added cars from Follower1 - Wait Until Keyword Succeeds 60s 2s Get Cars And Verify @{CAR_FOLLOWERS}[0] ${NUM_ENTRIES} + Get Cars And Verify @{CAR_FOLLOWERS}[0] ${NUM_ENTRIES} -Get added cars from Follower2 +Get Added Cars From Follower2 [Documentation] Get added cars from Follower2 - Wait Until Keyword Succeeds 60s 2s Get Cars And Verify @{CAR_FOLLOWERS}[1] ${NUM_ENTRIES} + Get Cars And Verify @{CAR_FOLLOWERS}[1] ${NUM_ENTRIES} Get People Leader And Followers - ${CURRENT_PEOPLE_LEADER} Wait For Leader To Be Found ${SHARD_PEOPLE_NAME} + [Documentation] Find leader and followers in the people shard + ${CURRENT_PEOPLE_LEADER} Get Leader And Verify ${SHARD_PEOPLE_NAME} Set Suite Variable ${CURRENT_PEOPLE_LEADER} ${PEOPLE_FOLLOWERS} Get All Followers ${SHARD_PEOPLE_NAME} Set Suite Variable ${PEOPLE_FOLLOWERS} -Add people and get people from Leader +Add People And Get People From Leader [Documentation] Add some people and get people from Leader. Add People And Verify ${CURRENT_PEOPLE_LEADER} ${NUM_ENTRIES} -Get added people from Follower1 +Get Added People From Follower1 [Documentation] Get added people from Follower1 - Wait Until Keyword Succeeds 60s 2s Get People And Verify @{PEOPLE_FOLLOWERS}[0] ${NUM_ENTRIES} + Get People And Verify @{PEOPLE_FOLLOWERS}[0] ${NUM_ENTRIES} -Get added people from Follower2 +Get Added People From Follower2 [Documentation] Get added people from Follower2 - Wait Until Keyword Succeeds 60s 2s Get People And Verify @{PEOPLE_FOLLOWERS}[1] ${NUM_ENTRIES} + Get People And Verify @{PEOPLE_FOLLOWERS}[1] ${NUM_ENTRIES} Get Car-Person Leader And Followers - ${CURRENT_CAR_PERSON_LEADER} Wait For Leader To Be Found ${SHARD_CAR_PERSON_NAME} + [Documentation] Find leader and followers in the car-person shard + ${CURRENT_CAR_PERSON_LEADER} Get Leader And Verify ${SHARD_CAR_PERSON_NAME} Set Suite Variable ${CURRENT_CAR_PERSON_LEADER} ${CAR_PERSON_FOLLOWERS} Get All Followers ${SHARD_CAR_PERSON_NAME} Set Suite Variable ${CAR_PERSON_FOLLOWERS} -Add car-person mapping and get car-person mapping from Leader +Add Car-Person Mapping And Get Car-Person Mapping From Leader + [Documentation] Initialize car-person shard Add Car Person And Verify ${CURRENT_CAR_PERSON_LEADER} -Purchase cars on Leader +Purchase Cars On Leader [Documentation] Purchase some cars on the Leader ${NUM_BUY_CARS_ON_LEADER} Evaluate ${NUM_ENTRIES}/3 ${NUM_BUY_CARS_ON_FOLLOWER1} Evaluate ${NUM_ENTRIES}/3 @@ -73,23 +71,23 @@ Purchase cars on Leader Set Suite Variable ${NUM_BUY_CARS_ON_FOLLOWER2} Buy Cars And Verify ${CURRENT_CAR_PERSON_LEADER} ${NUM_BUY_CARS_ON_LEADER} -Purchase cars on Follower1 +Purchase Cars On Follower1 [Documentation] Purchase some cars on Follower1 Buy Cars And Verify @{CAR_PERSON_FOLLOWERS}[0] ${NUM_BUY_CARS_ON_FOLLOWER1} ${NUM_BUY_CARS_ON_LEADER} -Purchase cars on Follower2 +Purchase Cars On Follower2 [Documentation] Purchase some cars on Follower2 ${start} Evaluate ${NUM_BUY_CARS_ON_LEADER}+${NUM_BUY_CARS_ON_FOLLOWER1} Buy Cars And Verify @{CAR_PERSON_FOLLOWERS}[1] ${NUM_BUY_CARS_ON_FOLLOWER2} ${start} -Get car-person mappings from Leader +Get Car-Person Mappings From Leader [Documentation] Get car-person mappings from Leader to see all entries - Wait Until Keyword Succeeds 60s 2s Get Car-Person Mappings And Verify ${CURRENT_CAR_PERSON_LEADER} ${NUM_ENTRIES} + Get Car-Person Mappings And Verify ${CURRENT_CAR_PERSON_LEADER} ${NUM_ENTRIES} -Get car-person mappings from Follower1 +Get Car-Person Mappings From Follower1 [Documentation] Get car-person mappings from Follower1 to see all entries - Wait Until Keyword Succeeds 60s 2s Get Car-Person Mappings And Verify @{CAR_PERSON_FOLLOWERS}[0] ${NUM_ENTRIES} + Get Car-Person Mappings And Verify @{CAR_PERSON_FOLLOWERS}[0] ${NUM_ENTRIES} -Get car-person mappings from Follower2 +Get Car-Person Mappings From Follower2 [Documentation] Get car-person mappings from Follower2 to see all entries - Wait Until Keyword Succeeds 60s 2s Get Car-Person Mappings And Verify @{CAR_PERSON_FOLLOWERS}[1] ${NUM_ENTRIES} + Get Car-Person Mappings And Verify @{CAR_PERSON_FOLLOWERS}[1] ${NUM_ENTRIES} diff --git a/csit/suites/controller/Clustering_Datastore/020_crud_on_any_follower.robot b/csit/suites/controller/Clustering_Datastore/020_crud_on_any_follower.robot index abe66b228a..aa8ff496f2 100644 --- a/csit/suites/controller/Clustering_Datastore/020_crud_on_any_follower.robot +++ b/csit/suites/controller/Clustering_Datastore/020_crud_on_any_follower.robot @@ -1,13 +1,8 @@ *** Settings *** Documentation This test finds the followers of certain shards in a 3-Node cluster and executes CRUD operations on any one follower Default Tags 3-node-cluster -Library Collections -Library ../../../libraries/Common.py -Library ../../../libraries/CrudLibrary.py -Library ../../../libraries/SettingsLibrary.py -Library ../../../libraries/UtilLibrary.py -Library ../../../libraries/ClusterStateLibrary.py Resource ../../../libraries/ClusterKeywords.robot +Resource ../../../libraries/CarsAndPeople.robot Variables ../../../variables/Variables.py *** Variables *** @@ -18,10 +13,12 @@ ${NUM_ENTRIES} ${40} *** Test Cases *** Get Car Followers + [Documentation] Find followers in the car shard ${CAR_FOLLOWERS} Get All Followers ${SHARD_CAR_NAME} Set Suite Variable ${CAR_FOLLOWERS} Get People Followers + [Documentation] Find followers in the people shard ${PEOPLE_FOLLOWERS} Get All Followers ${SHARD_PEOPLE_NAME} Set Suite Variable ${PEOPLE_FOLLOWERS} @@ -29,47 +26,47 @@ Get Car-Person Followers ${CAR_PERSON_FOLLOWERS} Get All Followers ${SHARD_CAR_PERSON_NAME} Set Suite Variable ${CAR_PERSON_FOLLOWERS} -Delete cars from Follower1 +Delete Cars From Follower1 Delete All Cars And Verify @{CAR_FOLLOWERS}[0] -Delete people from Follower1 +Delete People From Follower1 Delete All People And Verify @{PEOPLE_FOLLOWERS}[0] -Delete car-persons from Follower1 +Delete Car-Persons from Follower1 Delete All Cars-Persons And Verify @{CAR_PERSON_FOLLOWERS}[0] -Add cars and get cars from Follower1 +Add Cars And Get Cars From Follower1 [Documentation] Add cars and get added cars from Follower1 Add Cars And Verify @{CAR_FOLLOWERS}[0] ${NUM_ENTRIES} -Get added cars from Follower2 +Get Added Cars From Follower2 [Documentation] Get added cars from Follower2 - Wait Until Keyword Succeeds 60s 2s Get Cars And Verify @{CAR_FOLLOWERS}[1] ${NUM_ENTRIES} + Get Cars And Verify @{CAR_FOLLOWERS}[1] ${NUM_ENTRIES} -Add people and get people from Follower1 +Add People And Get People From Follower1 [Documentation] Add people and get people from Follower1 Add People And Verify @{PEOPLE_FOLLOWERS}[0] ${NUM_ENTRIES} -Get added people from Follower2 +Get Added People From Follower2 [Documentation] Get added people from Follower2 - Wait Until Keyword Succeeds 60s 2s Get People And Verify @{PEOPLE_FOLLOWERS}[1] ${NUM_ENTRIES} + Get People And Verify @{PEOPLE_FOLLOWERS}[1] ${NUM_ENTRIES} -Add car-person mapping and get car-person mapping from Follower1 +Add Car-Person Mapping And Get Car-Person Mapping From Follower1 Add Car Person And Verify @{CAR_PERSON_FOLLOWERS}[0] -Purchase cars on Follower1 +Purchase Cars On Follower1 [Documentation] Purchase cars using Follower1 Buy Cars And Verify @{CAR_PERSON_FOLLOWERS}[0] ${NUM_ENTRIES} -Get car-person mappings from Follower1 +Get Car-Person Mappings From Follower1 [Documentation] Get car-person mappings from Follower1 to see all entries - Wait Until Keyword Succeeds 60s 2s Get Car-Person Mappings And Verify @{CAR_PERSON_FOLLOWERS}[0] ${NUM_ENTRIES} + Get Car-Person Mappings And Verify @{CAR_PERSON_FOLLOWERS}[0] ${NUM_ENTRIES} -Get car-person mappings from Leader +Get Car-Person Mappings From Leader [Documentation] Get car-person mappings from the Leader to see all entries - ${CURRENT_CAR_LEADER} Wait For Leader To Be Found ${SHARD_CAR_PERSON_NAME} - Wait Until Keyword Succeeds 60s 2s Get Car-Person Mappings And Verify ${CURRENT_CAR_LEADER} ${NUM_ENTRIES} + ${CURRENT_CAR_LEADER} Get Leader And Verify ${SHARD_CAR_PERSON_NAME} + Get Car-Person Mappings And Verify ${CURRENT_CAR_LEADER} ${NUM_ENTRIES} -Get car-person mappings from Follower2 +Get Car-Person Mappings From Follower2 [Documentation] Get car-person mappings from Follower2 to see all entries - Wait Until Keyword Succeeds 60s 2s Get Car-Person Mappings And Verify @{CAR_PERSON_FOLLOWERS}[1] ${NUM_ENTRIES} + Get Car-Person Mappings And Verify @{CAR_PERSON_FOLLOWERS}[1] ${NUM_ENTRIES} diff --git a/csit/suites/controller/Clustering_Datastore/030_car_failover_crud_on_new_leader.robot b/csit/suites/controller/Clustering_Datastore/030_car_failover_crud_on_new_leader.robot index 3d51975473..8c8f432668 100644 --- a/csit/suites/controller/Clustering_Datastore/030_car_failover_crud_on_new_leader.robot +++ b/csit/suites/controller/Clustering_Datastore/030_car_failover_crud_on_new_leader.robot @@ -2,65 +2,77 @@ Documentation This test brings down the current leader of the "car" shard and then executes CRUD ... operations on the new leader Default Tags 3-node-cluster -Library ../../../libraries/CrudLibrary.py -Library ../../../libraries/UtilLibrary.py -Library ../../../libraries/ClusterStateLibrary.py Resource ../../../libraries/ClusterKeywords.robot +Resource ../../../libraries/CarsAndPeople.robot +Variables ../../../variables/Variables.py *** Variables *** ${CAR_SHARD} shard-car-config ${NUM_CARS} ${50} ${NUM_ORIG_CARS} ${10} ${KARAF_HOME} ${WORKSPACE}/${BUNDLEFOLDER} +${START_TIMEOUT} 300s +${STOP_TIMEOUT} 180s *** Test Cases *** -Get old car leader - ${OLD_CAR_LEADER} Wait For Leader To Be Found ${CAR_SHARD} +Get Old Car Leader + [Documentation] Find leader in the car shard + ${OLD_CAR_LEADER} Get Leader And Verify ${CAR_SHARD} Set Suite Variable ${OLD_CAR_LEADER} -Delete cars on old leader +Delete Cars On Old Leader + [Documentation] Delete cars in Leader Delete All Cars And Verify ${OLD_CAR_LEADER} -Add original cars on old leader +Add Original Cars On Old Leader + [Documentation] Add new cars in Leader and verify Add Cars And Verify ${OLD_CAR_LEADER} ${NUM_ORIG_CARS} -Switch car leader +Switch Car Leader [Documentation] Stop the leader to cause a new leader to be elected - ${NEW_CAR_LEADER} Switch Leader ${CAR_SHARD} ${OLD_CAR_LEADER} + Stop One Or More Controllers ${OLD_CAR_LEADER} + Wait For Controller Down ${STOP_TIMEOUT} ${OLD_CAR_LEADER} + ${NEW_CAR_LEADER} Wait Until Keyword Succeeds 30s 2s Get Leader And Verify ${CAR_SHARD} ${OLD_CAR_LEADER} Set Suite Variable ${NEW_CAR_LEADER} -Get original cars on new leader - Wait Until Keyword Succeeds 60s 2s Get Cars And Verify ${NEW_CAR_LEADER} ${NUM_ORIG_CARS} +Get Original Cars On New Leader + [Documentation] Get cars in new Leader + Get Cars And Verify ${NEW_CAR_LEADER} ${NUM_ORIG_CARS} -Delete cars on new leader +Delete Cars On New Leader + [Documentation] Delete cars in new Leader Delete All Cars And Verify ${NEW_CAR_LEADER} -Add new cars and get cars from new leader +Add New Cars And Get Cars From New Leader [Documentation] Add cars and get added cars from the Leader Add Cars And Verify ${NEW_CAR_LEADER} ${NUM_CARS} Get Car Followers + [Documentation] Find followers in the car shard ${CAR_FOLLOWERS} Get All Followers ${CAR_SHARD} ${OLD_CAR_LEADER} Set Suite Variable ${CAR_FOLLOWERS} -Get added cars from Follower +Get Added Cars From Follower [Documentation] Get the added cars from the Follower - Wait Until Keyword Succeeds 60s 2s Get Cars And Verify @{CAR_FOLLOWERS}[0] ${NUM_CARS} + Get Cars And Verify @{CAR_FOLLOWERS}[0] ${NUM_CARS} -Delete cars on Follower +Delete Cars On Follower + [Documentation] Delete cars in follower Delete All Cars And Verify @{CAR_FOLLOWERS}[0] -Add cars from Follower +Add Cars From Follower [Documentation] Add more cars from the Follower Add Cars And Verify @{CAR_FOLLOWERS}[0] ${NUM_CARS} -Get added cars from new leader +Get Added Cars From New Leader [Documentation] Get added cars from the new leader - Wait Until Keyword Succeeds 60s 2s Get Cars And Verify ${NEW_CAR_LEADER} ${NUM_CARS} + Get Cars And Verify ${NEW_CAR_LEADER} ${NUM_CARS} -Restart old Car leader +Restart Old Car Leader + [Documentation] Start old car Leader Start One Or More Controllers ${OLD_CAR_LEADER} + Wait For Controller Sync ${START_TIMEOUT} ${OLD_CAR_LEADER} -Get added cars from old leader +Get Added Cars From Old Leader [Documentation] Get the added cars from the old leader - Wait Until Keyword Succeeds 60s 2s Get Cars And Verify ${OLD_CAR_LEADER} ${NUM_CARS} + Wait Until Keyword Succeeds ${START_TIMEOUT} 2s Get Cars And Verify ${OLD_CAR_LEADER} ${NUM_CARS} diff --git a/csit/suites/controller/Clustering_Datastore/040_people_failover_crud_on_new_leader.robot b/csit/suites/controller/Clustering_Datastore/040_people_failover_crud_on_new_leader.robot index e0ffefbfe7..326d2ba0b6 100644 --- a/csit/suites/controller/Clustering_Datastore/040_people_failover_crud_on_new_leader.robot +++ b/csit/suites/controller/Clustering_Datastore/040_people_failover_crud_on_new_leader.robot @@ -1,50 +1,68 @@ *** Settings *** Documentation This test brings down the current leader of the "car" shard and then executes CRUD operations on the new leader Default Tags 3-node-cluster -Library ../../../libraries/CrudLibrary.py -Library ../../../libraries/UtilLibrary.py -Library ../../../libraries/ClusterStateLibrary.py Resource ../../../libraries/ClusterKeywords.robot +Resource ../../../libraries/CarsAndPeople.robot +Variables ../../../variables/Variables.py *** Variables *** ${PEOPLE_SHARD} shard-people-config ${NUM_ENTRIES} ${50} ${KARAF_HOME} ${WORKSPACE}/${BUNDLEFOLDER} +${START_TIMEOUT} 300s +${STOP_TIMEOUT} 180s *** Test Cases *** +Get Old People Leader + [Documentation] Find leader in the people shard + ${OLD_PEOPLE_LEADER} Get Leader And Verify ${PEOPLE_SHARD} + Set Suite Variable ${OLD_PEOPLE_LEADER} + Switch People Leader [Documentation] Stop the leader to cause a new leader to be elected - ${OLD_PEOPLE_LEADER} Wait For Leader To Be Found ${PEOPLE_SHARD} - ${NEW_PEOPLE_LEADER} Switch Leader ${PEOPLE_SHARD} ${OLD_PEOPLE_LEADER} - Set Suite Variable ${OLD_PEOPLE_LEADER} + Stop One Or More Controllers ${OLD_PEOPLE_LEADER} + Wait For Controller Down ${STOP_TIMEOUT} ${OLD_PEOPLE_LEADER} + ${NEW_PEOPLE_LEADER} Wait Until Keyword Succeeds 30s 2s Get Leader And Verify ${PEOPLE_SHARD} ${OLD_PEOPLE_LEADER} Set Suite Variable ${NEW_PEOPLE_LEADER} -Delete people from new leader +Delete People From New Leader + [Documentation] Delete people in new Leader Delete All People And Verify ${NEW_PEOPLE_LEADER} -Add people and get from new leader +Add People And Get From New Leader [Documentation] Add people and get people from new leader Add People And Verify ${NEW_PEOPLE_LEADER} ${NUM_ENTRIES} Get People Followers + [Documentation] Find followers in the people shard ${PEOPLE_FOLLOWERS} Get All Followers ${PEOPLE_SHARD} ${OLD_PEOPLE_LEADER} Set Suite Variable ${PEOPLE_FOLLOWERS} -Get added people from Follower - Wait Until Keyword Succeeds 60s 2s Get People And Verify @{PEOPLE_FOLLOWERS}[0] ${NUM_ENTRIES} +Get Added People From Follower + [Documentation] Get people in follower and verify + Get People And Verify @{PEOPLE_FOLLOWERS}[0] ${NUM_ENTRIES} -Delete people from new Follower +Delete People From New Follower + [Documentation] Delete people in follower and verify Delete All People And Verify @{PEOPLE_FOLLOWERS}[0] -Add people from new Follower - [Documentation] Add people and get people from follower +Add People From New Follower + [Documentation] Add people in follower and verify Add People And Verify @{PEOPLE_FOLLOWERS}[0] ${NUM_ENTRIES} -Get added people from new leader - Wait Until Keyword Succeeds 60s 2s Get People And Verify ${NEW_PEOPLE_LEADER} ${NUM_ENTRIES} +Get Added People From New Leader + [Documentation] Get people in Leader and verify + Get People And Verify ${NEW_PEOPLE_LEADER} ${NUM_ENTRIES} -Restart old People leader +Restart Old People Leader + [Documentation] Start old people Leader Start One Or More Controllers ${OLD_PEOPLE_LEADER} + Wait For Controller Sync ${START_TIMEOUT} ${OLD_PEOPLE_LEADER} + +Check Cars In Old People Leader + [Documentation] Check cars in new Leader. This is to avoid delay when RPC does not work. + Wait Until Keyword Succeeds ${START_TIMEOUT} 2s Check Cars ${OLD_PEOPLE_LEADER} ${NUM_ENTRIES} -Get added people from old leader - Wait Until Keyword Succeeds 60s 2s Get People And Verify ${OLD_PEOPLE_LEADER} ${NUM_ENTRIES} +Get Added People From Old Leader + [Documentation] Get people in old Leader and verify + Get People And Verify ${OLD_PEOPLE_LEADER} ${NUM_ENTRIES} diff --git a/csit/suites/controller/Clustering_Datastore/050_car_persistence_recovery.robot b/csit/suites/controller/Clustering_Datastore/050_car_persistence_recovery.robot index c6cfba6e9c..6ffc97de8f 100644 --- a/csit/suites/controller/Clustering_Datastore/050_car_persistence_recovery.robot +++ b/csit/suites/controller/Clustering_Datastore/050_car_persistence_recovery.robot @@ -1,10 +1,8 @@ *** Settings *** Documentation This test restarts all controllers to verify recovery of car data from persistene Default Tags 3-node-cluster -Library ../../../libraries/CrudLibrary.py -Library ../../../libraries/UtilLibrary.py -Library ../../../libraries/ClusterStateLibrary.py Resource ../../../libraries/ClusterKeywords.robot +Resource ../../../libraries/CarsAndPeople.robot Variables ../../../variables/Variables.py *** Variables *** @@ -12,43 +10,71 @@ ${CAR_SHARD} shard-car-config ${NUM_CARS} ${50} ${KARAF_HOME} ${WORKSPACE}/${BUNDLEFOLDER} @{controllers} ${CONTROLLER} ${CONTROLLER1} ${CONTROLLER2} +${START_TIMEOUT} 300s +${STOP_TIMEOUT} 180s *** Test Cases *** -Get car leader - ${CAR_LEADER} Wait For Leader To Be Found ${CAR_SHARD} +Get Car Leader + [Documentation] Find leader in the car shard + ${CAR_LEADER} Get Leader And Verify ${CAR_SHARD} Set Suite Variable ${CAR_LEADER} -Delete cars from leader +Delete Cars From Leader + [Documentation] Delete cars in Leader Delete All Cars And Verify ${CAR_LEADER} -Stop all controllers after delete +Stop All Controllers After Delete + [Documentation] Stop all controllers Stop One Or More Controllers @{controllers} + Wait For Cluster Down ${STOP_TIMEOUT} @{controllers} -Start all controllers after delete +Start All Controllers After Delete + [Documentation] Start all controller Start One Or More Controllers @{controllers} + Wait For Cluster Sync ${START_TIMEOUT} @{controllers} -Verify no cars on leader after restart - ${resp} Getcars ${CAR_LEADER} ${RESTCONFPORT} ${0} - Should Be Equal As Strings ${resp.status_code} 404 +Get Car Leader After First Restart + [Documentation] Find leader in the car shard + ${CAR_LEADER} Get Leader And Verify ${CAR_SHARD} + Set Suite Variable ${CAR_LEADER} + +Verify No Cars On Leader After Restart + [Documentation] Verify no cars after restart + Wait Until Keyword Succeeds ${START_TIMEOUT} 2s Check Cars Deleted ${CAR_LEADER} -Add cars on leader +Add Cars On Leader + [Documentation] Add cars in Leader Add Cars And Verify ${CAR_LEADER} ${NUM_CARS} -Stop all controllers after add +Stop All Controllers After Add + [Documentation] Stop all controllers Stop One Or More Controllers @{controllers} + Wait For Cluster Down ${STOP_TIMEOUT} @{controllers} -Start all controllers after add +Start All Controllers After Add + [Documentation] Start all controllers Start One Or More Controllers @{controllers} + Wait For Cluster Sync ${START_TIMEOUT} @{controllers} -Get cars from leader after restart - Wait Until Keyword Succeeds 60s 2s Get Cars And Verify ${CAR_LEADER} ${NUM_CARS} +Get Car Leader After Second Restart + [Documentation] Find leader in the car shard + ${CAR_LEADER} Get Leader And Verify ${CAR_SHARD} + Set Suite Variable ${CAR_LEADER} -Get car followers +Get Cars From Leader After Restart + [Documentation] Get cars from Leader and verify + Wait Until Keyword Succeeds ${START_TIMEOUT} 2s Get Cars And Verify ${CAR_LEADER} ${NUM_CARS} + +Get Car Followers + [Documentation] Find followers in the car shard ${CAR_FOLLOWERS} Get All Followers ${CAR_SHARD} Set Suite Variable ${CAR_FOLLOWERS} -Get cars from Follower1 after restart - Wait Until Keyword Succeeds 60s 2s Get Cars And Verify @{CAR_FOLLOWERS}[0] ${NUM_CARS} +Get Cars From Follower1 After Restart + [Documentation] Get cars in follower and verify + Wait Until Keyword Succeeds ${START_TIMEOUT} 2s Get Cars And Verify @{CAR_FOLLOWERS}[0] ${NUM_CARS} + +Get Cars From Follower2 After Restart + [Documentation] Get cars in follower and verify + Wait Until Keyword Succeeds ${START_TIMEOUT} 2s Get Cars And Verify @{CAR_FOLLOWERS}[1] ${NUM_CARS} -Get cars from Follower2 after restart - Wait Until Keyword Succeeds 60s 2s Get Cars And Verify @{CAR_FOLLOWERS}[1] ${NUM_CARS} diff --git a/csit/suites/controller/Clustering_Datastore/140_recovery_restart_follower.robot b/csit/suites/controller/Clustering_Datastore/140_recovery_restart_follower.robot index a57141c371..472864fd38 100644 --- a/csit/suites/controller/Clustering_Datastore/140_recovery_restart_follower.robot +++ b/csit/suites/controller/Clustering_Datastore/140_recovery_restart_follower.robot @@ -3,12 +3,9 @@ Documentation This test kills any of the followers and verifies that when th Default Tags 3-node-cluster Library Collections Library RequestsLibrary -Library ../../../libraries/Common.py Library ../../../libraries/CrudLibrary.py -Library ../../../libraries/SettingsLibrary.py -Library ../../../libraries/UtilLibrary.py -Library ../../../libraries/ClusterStateLibrary.py Resource ../../../libraries/ClusterKeywords.robot +Resource ../../../libraries/CarsAndPeople.robot Variables ../../../variables/Variables.py *** Variables *** @@ -16,11 +13,14 @@ ${CAR_SHARD} shard-car-config ${NUM_CARS} ${60} @{controllers} ${CONTROLLER} ${CONTROLLER1} ${CONTROLLER2} ${KARAF_HOME} ${WORKSPACE}/${BUNDLEFOLDER} +${START_TIMEOUT} 300s +${STOP_TIMEOUT} 180s *** Test Cases *** Stop All Controllers [Documentation] Stop all the controllers in the cluster Stop One Or More Controllers @{controllers} + Wait For Cluster Down ${STOP_TIMEOUT} @{controllers} Clean All Journals [Documentation] Clean the journals of all the controllers in the cluster @@ -29,39 +29,48 @@ Clean All Journals Start All Controllers [Documentation] Start all the controllers in the cluster Start One Or More Controllers @{controllers} + Wait For Cluster Sync ${START_TIMEOUT} @{controllers} -Get car leader and followers - ${CURRENT_CAR_LEADER} Wait For Leader To Be Found ${CAR_SHARD} +Get Car Leader And Followers + [Documentation] Find leader and followers in the car shard + ${CURRENT_CAR_LEADER} Get Leader And Verify ${CAR_SHARD} Set Suite Variable ${CURRENT_CAR_LEADER} ${CAR_FOLLOWERS} Get All Followers ${CAR_SHARD} Set Suite Variable ${CAR_FOLLOWERS} -Stop both of the followers +Verify No Cars On Leader After Restart + [Documentation] Verify no cars after restart + Wait Until Keyword Succeeds ${START_TIMEOUT} 2s Check Cars Deleted ${CURRENT_CAR_LEADER} + +Stop Both Of The Followers + [Documentation] Stop car followers @{followers} = Create List @{CAR_FOLLOWERS}[0] @{CAR_FOLLOWERS}[1] Stop One Or More Controllers @{followers} + Wait For Cluster Down ${STOP_TIMEOUT} @{followers} -Attempt to add a car to the leader - [Documentation] Should fail as both followers are down +Attempt To Add A Car To The Leader + [Documentation] Add car should fail as both followers are down AddCar ${CURRENT_CAR_LEADER} ${RESTCONFPORT} ${1} 500 Sleep 2 ${resp} Getcars ${CURRENT_CAR_LEADER} ${RESTCONFPORT} ${1} Should Not Be Equal As Strings ${resp.status_code} 200 -Restart the first follower +Restart The First Follower + [Documentation] Start one follower Start One Or More Controllers @{CAR_FOLLOWERS}[0] + Wait For Controller Sync ${START_TIMEOUT} @{CAR_FOLLOWERS}[0] -Add cars to the first follower +Add Cars To The First Follower + [Documentation] Add cars to the follower and verify Log Adding ${NUM_CARS} cars to @{CAR_FOLLOWERS}[0] - Add Cars And Verify Without Init @{CAR_FOLLOWERS}[0] ${NUM_CARS} 4s + Wait Until Keyword Succeeds ${START_TIMEOUT} 2s Add Cars And Verify Without Init @{CAR_FOLLOWERS}[0] ${NUM_CARS} -Restart the second follower +Restart The Second Follower + [Documentation] Start another follower Start One Or More Controllers @{CAR_FOLLOWERS}[1] + Wait For Controller Sync ${START_TIMEOUT} @{CAR_FOLLOWERS}[1] -Get all the cars from the second follower - Get Cars And Verify @{CAR_FOLLOWERS}[1] ${NUM_CARS} +Get All The Cars From The Second Follower + [Documentation] Add cars to the follower and verify + Wait Until Keyword Succeeds ${START_TIMEOUT} 2s Get Cars And Verify @{CAR_FOLLOWERS}[1] ${NUM_CARS} -*** Keywords *** -Expect No Leader - ${leader} GetLeader ${CAR_SHARD} ${3} ${1} ${1} ${RESTCONFPORT} - ... ${CURRENT_CAR_LEADER} - Should Be Equal As Strings ${leader} None diff --git a/csit/suites/controller/Clustering_Routedrpc/023_routed_rpc_crud_test.robot b/csit/suites/controller/Clustering_Routedrpc/023_routed_rpc_crud_test.robot index b63b11d821..896bb9e2b7 100644 --- a/csit/suites/controller/Clustering_Routedrpc/023_routed_rpc_crud_test.robot +++ b/csit/suites/controller/Clustering_Routedrpc/023_routed_rpc_crud_test.robot @@ -2,11 +2,9 @@ Documentation Test suite for Routed RPC. Library Collections Library RequestsLibrary -Library ../../../libraries/Common.py Library ../../../libraries/CrudLibrary.py -Library ../../../libraries/SettingsLibrary.py -Library ../../../libraries/UtilLibrary.py Resource ../../../libraries/ClusterKeywords.robot +Resource ../../../libraries/CarsAndPeople.robot Variables ../../../variables/Variables.py *** Variables *** @@ -15,114 +13,118 @@ ${SHARD_CAR_NAME} shard-car-config ${SHARD_PEOPLE_NAME} shard-people-config ${SHARD_CAR_PERSON_NAME} shard-car-people-config ${KARAF_HOME} ${WORKSPACE}/${BUNDLEFOLDER} +${NUM_ENTRIES} ${100} +${START_TIMEOUT} 300s +${STOP_TIMEOUT} 180s *** Test Cases *** -Delete all entries from shards +Delete All Entries [Documentation] Make sure the shards are cleared for testing. - : FOR ${ip} IN @{controllers} - \ Delete All Cars And Verify ${ip} - : FOR ${ip} IN @{controllers} - \ Delete All People And Verify ${ip} - : FOR ${ip} IN @{controllers} - \ Delete All Cars-Persons And Verify ${ip} + Delete All Entries From Shards @{controllers} Get Car Leader And Followers - ${CURRENT_CAR_LEADER} Wait For Leader To Be Found ${SHARD_CAR_NAME} + [Documentation] Find leader and followers in the car shard + ${CURRENT_CAR_LEADER} Get Leader And Verify ${SHARD_CAR_NAME} Set Suite Variable ${CURRENT_CAR_LEADER} @{CAR_FOLLOWERS} Get All Followers ${SHARD_CAR_NAME} Set Suite Variable ${CAR_FOLLOWERS} -Add cars and get cars from Leader +Add Cars And Get Cars From Leader [Documentation] Add 100 cars and get added cars from Leader - ${resp}= InitCar ${CURRENT_CAR_LEADER} ${PORT} - ${resp}= AddCar ${CURRENT_CAR_LEADER} ${PORT} ${100} - ${resp}= Getcars ${CURRENT_CAR_LEADER} ${PORT} ${0} - Should Be Equal As Strings ${resp.status_code} 200 - Should Contain ${resp.content} manufacturer1 cars not added! - -Add persons and get persons from Leader - [Documentation] Add 100 persons and get persons Note: There should be one person added first to enable rpc - ${resp} AddPerson ${CURRENT_CAR_LEADER} ${PORT} ${0} - ${resp} AddPerson ${CURRENT_CAR_LEADER} ${PORT} ${100} - ${resp} GetPersons ${CURRENT_CAR_LEADER} ${PORT} ${0} - Should Be Equal As Strings ${resp.status_code} 200 - Should Contain ${resp.content} user5 people not added! - -Add car-person mapping and get car-person mapping from Follower1 - [Documentation] Add car-person and get car-person from Leader Note: This is done to enable working of rpc - Log @{CAR_FOLLOWERS}[0] - ${resp} AddCarPerson @{CAR_FOLLOWERS}[0] ${PORT} ${0} - Should Be Equal As Strings ${resp.status_code} 204 - ${resp} GetCarPersonMappings @{CAR_FOLLOWERS}[0] ${PORT} ${0} - Should Be Equal As Strings ${resp.status_code} 200 - Should Contain ${resp.content} user0 car-person not initialized! + Add Cars And Verify ${CURRENT_CAR_LEADER} ${NUM_ENTRIES} + +Add Persons And Get Persons From Car Leader + [Documentation] Add 100 persons and get persons from Leader + Add People And Verify ${CURRENT_CAR_LEADER} ${NUM_ENTRIES} + +Add Car-Person Mapping And Get Car-Person Mapping From Car Follower1 + [Documentation] Add car-person and get car-person from Leader + Add Car Person And Verify @{CAR_FOLLOWERS}[0] -Purchase 100 cars using Follower1 +Purchase 100 Cars Using Car Follower1 [Documentation] Purchase 100 cars using Follower1 - BuyCar @{CAR_FOLLOWERS}[0] ${PORT} ${100} + Buy Cars And Verify @{CAR_FOLLOWERS}[0] ${NUM_ENTRIES} -Check Contents of Leader Shards +Check Contents Of Car Leader Shards [Documentation] Check all content using Leader - wait until keyword succeeds 30 1 Check Cars ${CURRENT_CAR_LEADER} ${PORT} 100 - wait until keyword succeeds 30 1 Check People ${CURRENT_CAR_LEADER} ${PORT} 100 - wait until keyword succeeds 30 1 Check CarPeople ${CURRENT_CAR_LEADER} ${PORT} 100 + Check Elements In Shards ${CURRENT_CAR_LEADER} ${NUM_ENTRIES} -Check Contents of First Follower Shards +Check Contents Of Car Follower1 Shards [Documentation] Check all content using first follower - wait until keyword succeeds 30 1 Check Cars @{CAR_FOLLOWERS}[0] ${PORT} 100 - wait until keyword succeeds 30 1 Check People @{CAR_FOLLOWERS}[0] ${PORT} 100 - wait until keyword succeeds 30 1 Check CarPeople @{CAR_FOLLOWERS}[0] ${PORT} 100 + Check Elements In Shards @{CAR_FOLLOWERS}[0] ${NUM_ENTRIES} -Check Contents of Second Follower Shards +Check Contents Of Car Follower2 Shards [Documentation] Check all content using second follower - wait until keyword succeeds 30 1 Check Cars @{CAR_FOLLOWERS}[1] ${PORT} 100 - wait until keyword succeeds 30 1 Check People @{CAR_FOLLOWERS}[1] ${PORT} 100 - wait until keyword succeeds 30 1 Check CarPeople @{CAR_FOLLOWERS}[1] ${PORT} 100 + Check Elements In Shards @{CAR_FOLLOWERS}[1] ${NUM_ENTRIES} -Switch Car leader - [Documentation] Stop the leader to cause a new leader to be elected - ${OLD_CAR_LEADER}= Set Variable ${CURRENT_CAR_LEADER} +Get Old Car Leader + [Documentation] Find leader in the car shard + ${OLD_CAR_LEADER} Get Leader And Verify ${SHARD_CAR_NAME} Set Suite Variable ${OLD_CAR_LEADER} - ${NEW_CAR_LEADER} Switch Leader ${SHARD_CAR_NAME} ${CURRENT_CAR_LEADER} + +Switch Car Leader + [Documentation] Stop the leader to cause a new leader to be elected + Stop One Or More Controllers ${OLD_CAR_LEADER} + Wait For Controller Down ${STOP_TIMEOUT} ${OLD_CAR_LEADER} + ${NEW_CAR_LEADER} Wait Until Keyword Succeeds 30s 2s Get Leader And Verify ${SHARD_CAR_NAME} ${OLD_CAR_LEADER} Set Suite Variable ${NEW_CAR_LEADER} -Get New Car Follower - [Documentation] Find the new leader for the car shard. +Get New Car Followers + [Documentation] Find the new followers for the car shard. @{CAR_FOLLOWERS} Get All Followers ${SHARD_CAR_NAME} Set Suite Variable ${CAR_FOLLOWERS} Log @{CAR_FOLLOWERS}[0] -Overwrite cars and get cars from New Leader - [Documentation] Overwrite 100 cars and get added cars from Leader - ${resp}= AddCar ${NEW_CAR_LEADER} ${PORT} ${100} - ${resp}= Getcars ${NEW_CAR_LEADER} ${PORT} ${0} - Should Be Equal As Strings ${resp.status_code} 200 - Should Contain ${resp.content} manufacturer1 cars not added! - -Overwrite persons and get persons from New Leader - [Documentation] Overwrite 100 persons and get persons Note: There should be one person added first to enable rpc - ${resp} AddPerson ${NEW_CAR_LEADER} ${PORT} ${100} - ${resp} GetPersons ${NEW_CAR_LEADER} ${PORT} ${0} - Should Be Equal As Strings ${resp.status_code} 200 - Should Contain ${resp.content} user5 people not added! - -RePurchase 100 cars using New Follower1 - [Documentation] RePurchase 100 cars using Follower1 - BuyCar @{CAR_FOLLOWERS}[0] ${PORT} ${100} - -Check Contents of New Leader Shards - [Documentation] Check all content using Leader - Log ${NEW_CAR_LEADER} - wait until keyword succeeds 30 1 Check Cars ${NEW_CAR_LEADER} ${PORT} 100 - wait until keyword succeeds 30 1 Check People ${NEW_CAR_LEADER} ${PORT} 100 - wait until keyword succeeds 30 1 Check CarPeople ${NEW_CAR_LEADER} ${PORT} 100 +Check Cars In New Car Leader + [Documentation] Check cars in new Leader + Wait Until Keyword Succeeds ${START_TIMEOUT} 2s Check Cars ${NEW_CAR_LEADER} ${NUM_ENTRIES} + +Check Contents Of New Car Leader Shards + [Documentation] Check all content using new Leader + Check Elements In Shards ${NEW_CAR_LEADER} ${NUM_ENTRIES} + +Check Contents Of New Car Follower1 Shards + [Documentation] Check all content using first follower + Check Elements In Shards @{CAR_FOLLOWERS}[0] ${NUM_ENTRIES} -Check Contents of New First Follower Shards +Readd People From New Car Leader + [Documentation] Add 100 persons and get persons from Leader + Add People And Verify Without Init ${NEW_CAR_LEADER} ${NUM_ENTRIES} + +Repurchase 100 Cars Using New Car Follower1 + [Documentation] Repurchase 100 cars using Follower1 + Buy Cars And Verify @{CAR_FOLLOWERS}[0] ${NUM_ENTRIES} + +Check Contents Of New Car Leader Shards After Repurchase + [Documentation] Check all content using new Leader + Check Elements In Shards ${NEW_CAR_LEADER} ${NUM_ENTRIES} + +Check Contents Of New Car First Follower Shards After Repurchase [Documentation] Check all content using first follower - wait until keyword succeeds 30 1 Check Cars @{CAR_FOLLOWERS}[0] ${PORT} 100 - wait until keyword succeeds 30 1 Check People @{CAR_FOLLOWERS}[0] ${PORT} 100 - wait until keyword succeeds 30 1 Check CarPeople @{CAR_FOLLOWERS}[0] ${PORT} 100 + Check Elements In Shards @{CAR_FOLLOWERS}[0] ${NUM_ENTRIES} -Start Leader +Start Old Car Leader [Documentation] Start Leader controller Start One Or More Controllers ${OLD_CAR_LEADER} + Wait For Controller Sync ${START_TIMEOUT} ${OLD_CAR_LEADER} + +Check Cars In Old Car Leader + [Documentation] Check cars in old Leader + Wait Until Keyword Succeeds ${START_TIMEOUT} 2s Check Cars ${OLD_CAR_LEADER} ${NUM_ENTRIES} + +Check Contents Of Old Leader Shards + [Documentation] Check all content using old Leader + Check Elements In Shards ${OLD_CAR_LEADER} ${NUM_ENTRIES} + +Readd People From Old Leader + [Documentation] Add 100 persons and get persons from Leader + Wait Until Keyword Succeeds 30 2s Add People And Verify Without Init ${OLD_CAR_LEADER} ${NUM_ENTRIES} + +Repurchase 100 Cars Using Old Leader + [Documentation] Repurchase 100 cars using Follower1 + Buy Cars And Verify ${OLD_CAR_LEADER} ${NUM_ENTRIES} + +Check Contents Of Old Leader Shards After Repurchase + [Documentation] Check all content using Leader + Check Elements In Shards ${OLD_CAR_LEADER} ${NUM_ENTRIES} + -- 2.36.6