2 Documentation Cluster suite for testing minimal and sum-minimal member population behavior.
4 ... Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.
6 ... This program and the accompanying materials are made available under the
7 ... terms of the Eclipse Public License v1.0 which accompanies this distribution,
8 ... and is available at http://www.eclipse.org/legal/epl-v10.html
11 ... This test kills majority of the followers and verifies car addition is not possible,
12 ... then resumes single follower (first from original list) and checks that addition works.
13 ... Then remaining members are brought up.
14 ... Leader member is always up and assumed to remain Leading during the whole suite run.
16 ... TODO: Use initial data to check more operations.
17 ... TODO: Perhaps merge with car_failover_crud suite.
19 ... Other modules and Shards (people, car-people) are not accessed by this suite.
21 ... All data is deleted at the end of the suite.
22 ... This suite expects car module to have a separate Shard.
24 Test Setup SetupUtils.Setup_Test_With_Logging_And_Without_Fast_Failing
25 Default Tags clustering carpeople critical
27 Resource ${CURDIR}/../../../libraries/CarPeople.robot
28 Resource ${CURDIR}/../../../libraries/ClusterManagement.robot
29 Resource ${CURDIR}/../../../libraries/SetupUtils.robot
30 Resource ${CURDIR}/../../../libraries/TemplatedRequests.robot
31 Variables ${CURDIR}/../../../variables/Variables.py
35 ${MINORITY_START_I} 300
36 ${MAJORITY_START_I} 200
37 ${MEMBER_START_TIMEOUT} 300s
38 @{SHARD_NAME_LIST} car
39 ${VAR_DIR} ${CURDIR}/../../../variables/carpeople/crud
40 ${CLUSTER_DIR} ${CURDIR}/../../../variables/clustering
43 Kill_Majority_Of_The_Followers
44 [Documentation] Kill half plus one car Follower members and set reviving followers down (otherwsise tipping followers cannot join cluster).
45 ... Mark most of killed members as explicitly down, to allow the surviving leader make progress.
46 ClusterManagement.Kill_Members_From_List_Or_All member_index_list=${list_of_killing} confirm=True
47 : FOR ${index} IN @{list_of_reviving}
48 \ ${data} OperatingSystem.Get File ${CLUSTER_DIR}/member_down.json
49 \ ${member_ip} = Collections.Get_From_Dictionary ${ClusterManagement__index_to_ip_mapping} ${index}
50 \ ${data} String.Replace String ${data} {member_ip} ${member_ip}
51 \ TemplatedRequests.Post_To_Uri uri=jolokia data=${data} content_type=${HEADERS} accept=${ACCEPT_EMPTY} session=${car_leader_session}
53 Attempt_To_Add_Cars_To_Leader
54 [Documentation] Adding cars should fail, as majority of Followers are down.
55 ${status} ${message} = BuiltIn.Run_Keyword_And_Ignore_Error TemplatedRequests.Put_As_Json_Templated folder=${VAR_DIR}/cars session=${car_leader_session} iterations=${CAR_ITEMS}
56 ... iter_start=${MINORITY_START_I}
57 # TODO: Is there a specific status and mesage to require in this scenario?
58 # Previously it was 500, now the restconf call simply times out.
59 BuiltIn.Log ${message}
60 BuiltIn.Should_Be_Equal ${status} FAIL
62 Start_Tipping_Follower
63 [Documentation] Start one Follower member without persisted data.
64 ClusterManagement.Start_Members_From_List_Or_All member_index_list=${list_of_tipping} wait_for_sync=True timeout=${MEMBER_START_TIMEOUT}
65 BuiltIn.Wait_Until_Keyword_Succeeds 30s 2s ClusterManagement.Verify_Leader_Exists_For_Each_Shard shard_name_list=${SHARD_NAME_LIST} shard_type=config member_index_list=${list_of_majority}
67 Add_Cars_On_Tipping_Follower
68 [Documentation] Add cars on the tipping Follower.
69 TemplatedRequests.Put_As_Json_Templated folder=${VAR_DIR}/cars session=${car_first_follower_session} iterations=${CAR_ITEMS} iter_start=${MAJORITY_START_I}
71 See_Cars_On_Existing_Members
72 [Documentation] On each up member: GET cars, should match the ones added on tipping Follower.
73 : FOR ${session} IN @{list_of_majority}
74 \ TemplatedRequests.Get_As_Json_Templated folder=${VAR_DIR}/cars session=${session} verify=True iterations=${CAR_ITEMS} iter_start=${MAJORITY_START_I}
77 [Documentation] Start other followers without persisted data.
78 ClusterManagement.Start_Members_From_List_Or_All member_index_list=${list_of_reviving} wait_for_sync=True timeout=${MEMBER_START_TIMEOUT}
79 BuiltIn.Wait_Until_Keyword_Succeeds 30s 2s ClusterManagement.Verify_Leader_Exists_For_Each_Shard shard_name_list=${SHARD_NAME_LIST} shard_type=config
81 See_Cars_On_New_Follower_Leader
82 [Documentation] GET cars from a new follower to see that the current state was replicated.
83 TemplatedRequests.Get_As_Json_Templated folder=${VAR_DIR}/cars session=${car_last_follower_session} verify=True iterations=${CAR_ITEMS} iter_start=${MAJORITY_START_I}
86 [Documentation] Delete cars on Leader.
87 TemplatedRequests.Delete_Templated folder=${VAR_DIR}/cars session=${car_leader_session}
91 [Documentation] Initialize resources, memorize shard leaders, pre-compute member lists.
92 SetupUtils.Setup_Utils_For_Setup_And_Teardown http_timeout=5
93 CarPeople.Set_Variables_For_Shard shard_name=car
94 Set_Additional_Variables
96 Set_Additional_Variables
97 [Documentation] Compute various lists useful for test cases in this suite.
98 # TODO: Migrate this Keyword to CarPeople Resource if more suites want that.
99 ${last_follower_session} = Collections.Get_From_List ${car_follower_sessions} -1
100 BuiltIn.Set_Suite_Variable \${car_last_follower_session} ${last_follower_session}
101 ${number_followers} = BuiltIn.Get_Length ${car_follower_indices}
102 ${half_followers} = BuiltIn.Evaluate ${number_followers} / 2
103 ${majority_follower_list} = Collections.Get_Slice_From_List ${car_follower_indices} 0 ${half_followers}
104 ${leader_list} = BuiltIn.Create_List ${car_leader_index}
105 ${majority_list} = Collections.Combine_Lists ${leader_list} ${majority_follower_list}
106 BuiltIn.Set_Suite_Variable \${list_of_majority} ${majority_list}
107 ${tipping_list} = Collections.Get_Slice_From_List ${majority_follower_list} 0 1
108 BuiltIn.Set_Suite_Variable \${list_of_tipping} ${tipping_list}
109 ${revive_list} = Collections.Get_Slice_From_List ${car_follower_indices} ${half_followers} ${number_followers}
110 BuiltIn.Set_Suite_Variable \${list_of_reviving} ${revive_list}
111 ${kill_list} = Collections.Combine_Lists ${tipping_list} ${revive_list}
112 BuiltIn.Set_Suite_Variable \${list_of_killing} ${kill_list}