2 Documentation Cluster Singleton testing: Common Keywords
4 ... Copyright (c) 2017 Cisco Systems, Inc. and others. All rights reserved.
6 ... This program and the accompanying materials are made available under the
7 ... terms of the Eclipse Public License v1.0 which accompanies this distribution,
8 ... and is available at http://www.eclipse.org/legal/epl-v10.html
10 ... Cluster Singleton service is designed to ensure that only one instance of an
11 ... application is registered globally in the cluster.
13 ... Creates and uses the following suite variables:
14 ... Created by Cluster_Singleton_Init:
16 ... ${cs_exp_candidates}
17 ... Created by Get_And_Save_Present_CsOwner_And_CsCandidates:
20 ... Created by Isolate_Owner_And_Verify_Isolated
21 ... ${cs_isolated_index}
24 Resource ${CURDIR}/../ClusterManagement.robot
25 Resource ${CURDIR}/../MdsalLowlevel.robot
26 Resource ${CURDIR}/../ShardStability.robot
27 Resource ${CURDIR}/../WaitForFailure.robot
31 ${CS_DEVICE_NAME} get-singleton-constant-service']
32 ${CS_DEVICE_TYPE} org.opendaylight.mdsal.ServiceEntityType
33 ${CS_CONSTANT_PREFIX} constant-
37 Cluster_Singleton_Init
38 [Documentation] Resouce initial keyword. Creates {cs_exp_candidates} and {cs_all_indices} suite variables which are
39 ... used in other keywords.
40 ${cs_exp_candidates} = BuiltIn.Create_List
41 BuiltIn.Set_Suite_Variable ${cs_exp_candidates}
42 ${cs_all_indices} = ClusterManagement.List_All_Indices
43 BuiltIn.Set_Suite_Variable ${cs_all_indices}
45 Register_Singleton_And_Update_Expected_Candidates
46 [Documentation] Register the singleton candidate and add it to the list of ${cs_exp_candidates}.
47 [Arguments] ${member_index} ${constant}
48 MdsalLowlevel.Register_Singleton_Constant ${member_index} ${constant}
49 Collections.Append_To_List ${cs_exp_candidates} ${member_index}
50 Collections.Sort_List ${cs_exp_candidates}
52 Unregister_Singleton_And_Update_Expected_Candidates
53 [Documentation] Unregister the singleton candidate. Also remove it from the list of ${cs_exp_candidates}.
54 [Arguments] ${member_index}
55 MdsalLowlevel.Unregister_Singleton_Constant ${member_index}
56 Collections.Remove_Values_From_List ${cs_exp_candidates} ${member_index}
58 Verify_Owner_And_Candidates_Stable
59 [Documentation] Fail if the actual owner is different from ${owner_index} or if the actual candidate list is different from ${cs_exp_candidates}.
60 [Arguments] ${owner_index}
61 ${actual_owner} ${actual_candidates} = ClusterManagement.Check_Old_Owner_Stays_Elected_For_Device
66 Collections.Lists_Should_Be_Equal ${cs_exp_candidates} ${actual_candidates}
68 Monitor_Owner_And_Candidates_Stability
69 [Documentation] Verify that the owner remains on the same node after non-owner candidate is unregistered.
70 [Arguments] ${monitoring_duration} ${owner_index}
71 WaitForFailure.Verify_Keyword_Does_Not_Fail_Within_Timeout
72 ... ${monitoring_duration}
74 ... Verify_Owner_And_Candidates_Stable
77 Register_Singleton_Constant_On_Nodes
78 [Documentation] Register a candidate application on given nodes.
79 [Arguments] ${index_list}
80 FOR ${index} IN @{index_list}
81 Register_Singleton_And_Update_Expected_Candidates ${index} ${CS_CONSTANT_PREFIX}${index}
84 Unregister_Singleton_Constant_On_Nodes
85 [Documentation] Unregister the application from given nodes.
86 [Arguments] ${index_list}
87 FOR ${index} IN @{index_list}
88 Unregister_Singleton_And_Update_Expected_Candidates ${index}
91 Get_And_Save_Present_CsOwner_And_CsCandidates
92 [Documentation] Store owner index into suite variables.
93 [Arguments] ${node_to_ask}
94 ${cs_owner} ${cs_candidates} = ClusterManagement.Get_Owner_And_Candidates_For_Device
98 BuiltIn.Set_Suite_Variable ${cs_owner}
99 BuiltIn.Set_Suite_Variable ${cs_candidates}
100 RETURN ${cs_owner} ${cs_candidates}
102 Verify_Singleton_Constant_On_Node
103 [Documentation] Verify that the expected constant is return from the given node.
104 [Arguments] ${node_to_ask} ${cs_exp_constant}
105 ${constant} = MdsalLowlevel.Get_Singleton_Constant ${node_to_ask}
106 BuiltIn.Should_Be_Equal ${cs_exp_constant} ${constant}
108 Verify_Singleton_Constant_On_Nodes
109 [Documentation] Iterate over all cluster nodes and all should return expected constant.
110 [Arguments] ${index_list} ${cs_exp_constant}
111 FOR ${index} IN @{index_list}
112 Verify_Singleton_Constant_On_Node ${index} ${cs_exp_constant}
115 Verify_Singleton_Constant_During_Isolation
116 [Documentation] Iterate over all non-isolated cluster nodes. They should return the correct constant.
117 FOR ${index} IN @{cs_all_indices}
118 IF "${index}" == "${cs_isolated_index}"
119 BuiltIn.Log Node not triggered, behavior not well described, see bugs 8207, 8214.
121 IF "${index}" != "${cs_isolated_index}"
122 Verify_Singleton_Constant_On_Node ${index} ${CS_CONSTANT_PREFIX}${cs_owner}
126 Isolate_Owner_And_Verify_Isolated
127 [Documentation] Isolate the owner cluster node. Wait until the new owner is elected and store new values of owner and candidates.
128 ClusterManagement.Isolate_Member_From_List_Or_All ${cs_owner}
129 BuiltIn.Set_Suite_Variable ${cs_isolated_index} ${cs_owner}
130 ${non_isolated_list} = ClusterManagement.List_Indices_Minus_Member
131 ... ${cs_isolated_index}
132 ... member_index_list=${cs_all_indices}
133 ${node_to_ask} = Collections.Get_From_list ${non_isolated_list} 0
134 BuiltIn.Wait_Until_Keyword_Succeeds
137 ... ShardStability.Shards_Stability_Get_Details
138 ... ${DEFAULT_SHARD_LIST}
139 ... member_index_list=${non_isolated_list}
140 BuiltIn.Wait_Until_Keyword_Succeeds
143 ... ClusterManagement.Check_New_Owner_Got_Elected_For_Device
144 ... ${CS_DEVICE_NAME}
145 ... ${CS_DEVICE_TYPE}
146 ... ${cs_isolated_index}
149 Get_And_Save_Present_CsOwner_And_CsCandidates ${node_to_ask}
150 BuiltIn.Wait_Until_Keyword_Succeeds 60s 3s Verify_Singleton_Constant_During_Isolation
152 Rejoin_Node_And_Verify_Rejoined
153 [Documentation] Rejoin isolated node.
154 ClusterManagement.Rejoin_Member_From_List_Or_All ${cs_isolated_index}
155 BuiltIn.Wait_Until_Keyword_Succeeds
158 ... ShardStability.Shards_Stability_Get_Details
159 ... ${DEFAULT_SHARD_LIST}
161 Register_Flapping_Singleton_On_Nodes
162 [Documentation] Register a candidate application on each node which starts the test.
163 [Arguments] ${index_list}
164 FOR ${index} IN @{index_list}
165 MdsalLowlevel.Register_Flapping_Singleton ${index}
168 Unregister_Flapping_Singleton_On_Nodes_And_Validate_Results
169 [Documentation] Unregister the testing service and check recevied statistics.
170 [Arguments] ${index_list} ${rate_limit_to_pass} ${test_duration}
171 ${movements_count} = BuiltIn.Set_Variable ${0}
172 FOR ${index} IN @{index_list}
173 ${count} = MdsalLowlevel.Unregister_Flapping_Singleton ${index}
175 BuiltIn.Fail No failure should have occured during the ${test_duration} timeout.
177 ${movements_count} = BuiltIn.Evaluate ${movements_count}+${count}
179 ${seconds} = DateTime.Convert_Time ${test_duration}
180 ${rate} = BuiltIn.Evaluate ${movements_count}/${seconds}
181 IF ${rate} < ${rate_limit_to_pass}
182 BuiltIn.Fail Acceptance rate ${rate_limit_to_pass} not reached, actual rate is ${rate}.