2 Documentation DOMRpcBroker testing: RPC Provider Partition And Heal
4 ... Copyright (c) 2017 Cisco Systems, Inc. and others. All rights reserved.
6 ... This program and the accompanying materials are made available under the
7 ... terms of the Eclipse Public License v1.0 which accompanies this distribution,
8 ... and is available at http://www.eclipse.org/legal/epl-v10.html
10 ... This tests establishes that the RPC service operates correctly when faced
11 ... with node failures.
12 ... This suite supports more than three node cluster setup too.
14 Suite Teardown SSHLibrary.Close_All_Connections
15 Test Setup SetupUtils.Setup_Test_With_Logging_And_Without_Fast_Failing
16 Test Teardown SetupUtils.Teardown_Test_Show_Bugs_If_Test_Failed
20 Resource ${CURDIR}/../../../libraries/MdsalLowlevel.robot
21 Resource ${CURDIR}/../../../libraries/SetupUtils.robot
22 Resource ${CURDIR}/../../../libraries/TemplatedRequests.robot
23 Resource ${CURDIR}/../../../libraries/WaitForFailure.robot
26 @{INSTALLED_RPC_MEMEBER_IDX_LIST} ${1} ${2}
27 ${TESTED_MEMBER_WITHOUT_RPC_IDX} ${3}
28 ${CONSTANT_PREFIX} member-
29 @{NON_WORKING_RPC_STATUS_CODE} ${501}
32 Register_Rpc_On_Two_Nodes
33 [Documentation] Register rpc on two nodes of the odl cluster.
34 : FOR ${index} IN @{INSTALLED_RPC_MEMEBER_IDX_LIST}
35 \ MdsalLowlevel.Register_Constant ${index} ${CONSTANT_PREFIX}${index}
37 Invoke_Rpc_On_Each_Node
38 [Documentation] Invoke get-constant rpc on every node of the cluster. When requested on the node with
39 ... local instance the local value is expected. If invoked on the node with no local instance, any remote
40 ... value is expected. From the constant returned from the ${TESTED_MEMBER_WITHOUT_RPC_IDX} node (with no rpc instance) an index of
41 ... the node to be isolated is derived. And in the tc Invoke_Rpc_On_Remaining_Nodes a different constant
42 ... is expected. The second for loop makes the suite suitable for more that 3 nodes cluster.
43 : FOR ${index} IN @{INSTALLED_RPC_MEMEBER_IDX_LIST}
44 \ Verify_Local_Rpc_Invoked ${index}
45 : FOR ${index} IN @{non_installed_rpc_member_idx_list}
46 \ ${constant} = Verify_Any_Remote_Rpc_Invoked ${index}
47 \ BuiltIn.Run_Keyword_If "${index}" == "${TESTED_MEMBER_WITHOUT_RPC_IDX}" BuiltIn.Set_Suite_Variable ${initial_const_on_tested_non_rpc_member} ${constant}
48 ${isolated_idx} = String.Replace_String ${initial_const_on_tested_non_rpc_member} ${CONSTANT_PREFIX} ${EMPTY}
49 BuiltIn.Set_Suite_Variable ${isolated_idx} ${${isolated_idx}}
52 [Documentation] Isolate one node with registered rpc.
53 ClusterManagement.Isolate_Member_From_List_Or_All ${isolated_idx}
55 Invoke_Rpc_On_Isolated_Node
56 [Documentation] Invoke rpc on isolated node. Because rpc is registered on this node, local constant
58 BuiltIn.Wait_Until_Keyword_Succeeds 3x 2s Verify_Local_Rpc_Invoked ${isolated_idx}
60 Invoke_Rpc_On_Remaining_Nodes
61 [Documentation] Invoke rpc on non-islolated nodes. As the only instance of rpc remained in the non-isolated
62 ... cluster nodes, only this value is expected.
63 ${index_list} = ClusterManagement.List_Indices_Minus_Member ${isolated_idx} ${all_indices}
64 : FOR ${index} IN @{index_list}
65 \ ${constant} = Verify_Any_Remote_Rpc_Invoked ${index}
66 \ BuiltIn.Should_Not_Be_Equal_As_Strings ${CONSTANT_PREFIX}${isolated_idx} ${constant}
68 Rejoin_Isolated_Member
69 [Documentation] Rejoin isolated node
70 ClusterManagement.Rejoin_Member_From_List_Or_All ${isolated_idx}
72 Invoke_Rpc_On_Each_Node_Again
73 [Documentation] Invoke rpc get-constant on every node. When requested on the node with
74 ... local instance the local value is expected. If invoked on the node with no local instance, any remote
75 ... value is expected.
76 : FOR ${index} IN @{all_indices}
77 \ BuiltIn.Run_Keyword_If ${index} in ${INSTALLED_RPC_MEMEBER_IDX_LIST} Verify_Local_Rpc_Invoked ${index}
78 \ BuiltIn.Run_Keyword_Unless ${index} in ${INSTALLED_RPC_MEMEBER_IDX_LIST} WaitForFailure.Verify_Keyword_Does_Not_Fail_Within_Timeout 20s 3s Verify_Any_Remote_Rpc_Invoked
81 Isolate_Member_Without_Registered_Rpc
82 [Documentation] Isolate one node with unregistered rpc.
83 ClusterManagement.Isolate_Member_From_List_Or_All ${TESTED_MEMBER_WITHOUT_RPC_IDX}
85 Verify_Rpc_Fails_On_Isolated_Member_Without_Rpc
86 [Documentation] Rpc should fail as it is requested on isolated node without rpc instance.
87 BuiltIn.Wait_Until_Keyword_Succeeds 15s 2s MdsalLowlevel.Get_Constant ${TESTED_MEMBER_WITHOUT_RPC_IDX} explicit_status_codes=${NON_WORKING_RPC_STATUS_CODE}
89 Rejoin_Isolated_Member_Without_Registered_Rpc
90 [Documentation] Rejoin isolated node.
91 ClusterManagement.Rejoin_Member_From_List_Or_All ${TESTED_MEMBER_WITHOUT_RPC_IDX}
93 Verify_Rpc_Again_Passes_On_Member_Without_Rpc
94 [Documentation] Verify rpc works after the node rejoin.
95 ${constant} = BuiltIn.Wait_Until_Keyword_Succeeds 10x 3s MdsalLowlevel.Get_Constant ${TESTED_MEMBER_WITHOUT_RPC_IDX}
96 Collections.List_Should_Contain_Value ${possible_constants} ${constant}
98 Unregister_Rpc_On_Each_Node
99 [Documentation] Inregister rpc on both nodes.
100 : FOR ${index} IN @{INSTALLED_RPC_MEMEBER_IDX_LIST}
101 \ MdsalLowlevel.Unregister_Constant ${index}
105 [Documentation] Setup keyword. Create ${possible_constants} list with possible variables of remote constants.
106 SetupUtils.Setup_Utils_For_Setup_And_Teardown
107 ${all_indices} = ClusterManagement.List_All_Indices
108 BuiltIn.Set_Suite_Variable ${all_indices}
109 ${non_installed_rpc_member_idx_list} = ClusterManagement.List_All_Indices
110 ${possible_constants} = BuiltIn.Create_List
111 : FOR ${index} IN @{INSTALLED_RPC_MEMEBER_IDX_LIST}
112 \ Collections.Append_To_List ${possible_constants} ${CONSTANT_PREFIX}${index}
113 \ ${non_installed_rpc_member_idx_list} = ClusterManagement.List_Indices_Minus_Member ${index} ${non_installed_rpc_member_idx_list}
114 BuiltIn.Set_Suite_Variable ${possible_constants}
115 BuiltIn.Set_Suite_Variable ${non_installed_rpc_member_idx_list}
117 Verify_Local_Rpc_Invoked
118 [Arguments] ${member_index}
119 [Documentation] Verify that local constant is received.
120 ${constant} = MdsalLowlevel.Get_Constant ${member_index}
121 BuiltIn.Should_Be_Equal_As_Strings ${CONSTANT_PREFIX}${member_index} ${constant}
122 BuiltIn.Return_From_Keyword ${constant}
124 Verify_Any_Remote_Rpc_Invoked
125 [Arguments] ${member_index}
126 [Documentation] Verify that any valid constant is received.
127 ${constant} = MdsalLowlevel.Get_Constant ${member_index}
128 Collections.List_Should_Contain_Value ${possible_constants} ${constant}
129 BuiltIn.Return_From_Keyword ${constant}