2 Documentation Cluster Singleton testing: Common Keywords
4 ... Copyright (c) 2017 Cisco Systems, Inc. and others. All rights reserved.
6 ... This program and the accompanying materials are made available under the
7 ... terms of the Eclipse Public License v1.0 which accompanies this distribution,
8 ... and is available at http://www.eclipse.org/legal/epl-v10.html
10 ... Cluster Singleton service is designed to ensure that only one instance of an
11 ... application is registered globally in the cluster.
13 ... Creates and uses the following suite variables:
14 ... Created by Cluster_Singleton_Init:
16 ... ${cs_exp_candidates}
17 ... Created by Get_And_Save_Present_CsOwner_And_CsCandidates:
20 ... Created by Isolate_Owner_And_Verify_Isolated
21 ... ${cs_isolated_index}
23 Resource ${CURDIR}/../ClusterManagement.robot
24 Resource ${CURDIR}/../MdsalLowlevel.robot
25 Resource ${CURDIR}/../ShardStability.robot
26 Resource ${CURDIR}/../WaitForFailure.robot
29 ${CS_DEVICE_NAME} get-singleton-constant-service']
30 ${CS_DEVICE_TYPE} org.opendaylight.mdsal.ServiceEntityType
31 ${CS_CONSTANT_PREFIX} constant-
34 Cluster_Singleton_Init
35 [Documentation] Resouce initial keyword. Creates {cs_exp_candidates} and {cs_all_indices} suite variables which are
36 ... used in other keywords.
37 ${cs_exp_candidates} = BuiltIn.Create_List
38 BuiltIn.Set_Suite_Variable ${cs_exp_candidates}
39 ${cs_all_indices} = ClusterManagement.List_All_Indices
40 BuiltIn.Set_Suite_Variable ${cs_all_indices}
42 Register_Singleton_And_Update_Expected_Candidates
43 [Arguments] ${member_index} ${constant}
44 [Documentation] Register the singleton candidate and add it to the list of ${cs_exp_candidates}.
45 MdsalLowlevel.Register_Singleton_Constant ${member_index} ${constant}
46 Collections.Append_To_List ${cs_exp_candidates} ${member_index}
47 Collections.Sort_List ${cs_exp_candidates}
49 Unregister_Singleton_And_Update_Expected_Candidates
50 [Arguments] ${member_index}
51 [Documentation] Unregister the singleton candidate. Also remove it from the list of ${cs_exp_candidates}.
52 MdsalLowlevel.Unregister_Singleton_Constant ${member_index}
53 Collections.Remove_Values_From_List ${cs_exp_candidates} ${member_index}
55 Verify_Owner_And_Candidates_Stable
56 [Arguments] ${owner_index}
57 [Documentation] Fail if the actual owner is different from ${owner_index} or if the actual candidate list is different from ${cs_exp_candidates}.
58 ${actual_owner} ${actual_candidates} ClusterManagement.Check_Old_Owner_Stays_Elected_For_Device ${CS_DEVICE_NAME} ${CS_DEVICE_TYPE} ${owner_index} ${owner_index}
59 Collections.Lists_Should_Be_Equal ${cs_exp_candidates} ${actual_candidates}
61 Monitor_Owner_And_Candidates_Stability
62 [Arguments] ${monitoring_duration} ${owner_index}
63 [Documentation] Verify that the owner remains on the same node after non-owner candidate is unregistered.
64 WaitForFailure.Verify_Keyword_Does_Not_Fail_Within_Timeout ${monitoring_duration} 3s Verify_Owner_And_Candidates_Stable ${owner_index}
66 Register_Singleton_Constant_On_Nodes
67 [Arguments] ${index_list}
68 [Documentation] Register a candidate application on given nodes.
69 FOR ${index} IN @{index_list}
70 Register_Singleton_And_Update_Expected_Candidates ${index} ${CS_CONSTANT_PREFIX}${index}
73 Unregister_Singleton_Constant_On_Nodes
74 [Arguments] ${index_list}
75 [Documentation] Unregister the application from given nodes.
76 FOR ${index} IN @{index_list}
77 Unregister_Singleton_And_Update_Expected_Candidates ${index}
80 Get_And_Save_Present_CsOwner_And_CsCandidates
81 [Arguments] ${node_to_ask}
82 [Documentation] Store owner index into suite variables.
83 ${cs_owner} ${cs_candidates} ClusterManagement.Get_Owner_And_Candidates_For_Device ${CS_DEVICE_NAME} ${CS_DEVICE_TYPE} ${node_to_ask}
84 BuiltIn.Set_Suite_Variable ${cs_owner}
85 BuiltIn.Set_Suite_Variable ${cs_candidates}
86 BuiltIn.Return_From_Keyword ${cs_owner} ${cs_candidates}
88 Verify_Singleton_Constant_On_Node
89 [Arguments] ${node_to_ask} ${cs_exp_constant}
90 [Documentation] Verify that the expected constant is return from the given node.
91 ${constant} = MdsalLowlevel.Get_Singleton_Constant ${node_to_ask}
92 BuiltIn.Should_Be_Equal ${cs_exp_constant} ${constant}
94 Verify_Singleton_Constant_On_Nodes
95 [Arguments] ${index_list} ${cs_exp_constant}
96 [Documentation] Iterate over all cluster nodes and all should return expected constant.
97 FOR ${index} IN @{index_list}
98 Verify_Singleton_Constant_On_Node ${index} ${cs_exp_constant}
101 Verify_Singleton_Constant_During_Isolation
102 [Documentation] Iterate over all non-isolated cluster nodes. They should return the correct constant.
103 FOR ${index} IN @{cs_all_indices}
104 BuiltIn.Run_Keyword_If "${index}" == "${cs_isolated_index}" BuiltIn.Log Node not triggered, behavior not well described, see bugs 8207, 8214.
105 BuiltIn.Run_Keyword_Unless "${index}" == "${cs_isolated_index}" Verify_Singleton_Constant_On_Node ${index} ${CS_CONSTANT_PREFIX}${cs_owner}
108 Isolate_Owner_And_Verify_Isolated
109 [Documentation] Isolate the owner cluster node. Wait until the new owner is elected and store new values of owner and candidates.
110 ClusterManagement.Isolate_Member_From_List_Or_All ${cs_owner}
111 BuiltIn.Set_Suite_Variable ${cs_isolated_index} ${cs_owner}
112 ${non_isolated_list} = ClusterManagement.List_Indices_Minus_Member ${cs_isolated_index} member_index_list=${cs_all_indices}
113 ${node_to_ask} = Collections.Get_From_list ${non_isolated_list} 0
114 BuiltIn.Wait_Until_Keyword_Succeeds 70s 10s ShardStability.Shards_Stability_Get_Details ${DEFAULT_SHARD_LIST} member_index_list=${non_isolated_list}
115 BuiltIn.Wait_Until_Keyword_Succeeds 30s 2s ClusterManagement.Check_New_Owner_Got_Elected_For_Device ${CS_DEVICE_NAME} ${CS_DEVICE_TYPE} ${cs_isolated_index}
116 ... ${node_to_ask} http_timeout=125
117 Get_And_Save_Present_CsOwner_And_CsCandidates ${node_to_ask}
118 BuiltIn.Wait_Until_Keyword_Succeeds 60s 3s Verify_Singleton_Constant_During_Isolation
120 Rejoin_Node_And_Verify_Rejoined
121 [Documentation] Rejoin isolated node.
122 ClusterManagement.Rejoin_Member_From_List_Or_All ${cs_isolated_index}
123 BuiltIn.Wait_Until_Keyword_Succeeds 70s 10s ShardStability.Shards_Stability_Get_Details ${DEFAULT_SHARD_LIST}
125 Register_Flapping_Singleton_On_Nodes
126 [Arguments] ${index_list}
127 [Documentation] Register a candidate application on each node which starts the test.
128 FOR ${index} IN @{index_list}
129 MdsalLowlevel.Register_Flapping_Singleton ${index}
132 Unregister_Flapping_Singleton_On_Nodes_And_Validate_Results
133 [Arguments] ${index_list} ${rate_limit_to_pass} ${test_duration}
134 [Documentation] Unregister the testing service and check recevied statistics.
135 ${movements_count} = BuiltIn.Set_Variable ${0}
136 FOR ${index} IN @{index_list}
137 ${count} = MdsalLowlevel.Unregister_Flapping_Singleton ${index}
138 BuiltIn.Run_Keyword_If ${count} < 0 BuiltIn.Fail No failure should have occured during the ${test_duration} timeout.
139 ${movements_count} = BuiltIn.Evaluate ${movements_count}+${count}
141 ${seconds} = DateTime.Convert_Time ${test_duration}
142 ${rate} = BuiltIn.Evaluate ${movements_count}/${seconds}
143 BuiltIn.Run_Keyword_If ${rate} < ${rate_limit_to_pass} BuiltIn.Fail Acceptance rate ${rate_limit_to_pass} not reached, actual rate is ${rate}.