... - Cluster Setup
... - Shard state, leader and followers
... - Entity Owner, candidates and successors
-... - Kill and Start Member
+... - Kill, Stop and Start Member
... - Isolate and Rejoin Member
... - Run Commands On Member
... - REST requests and checks on Members
${SINGLETON_NETCONF_DEVICE_ID_SUFFIX} ]]]]}']
${SINGLETON_ELECTION_ENTITY_TYPE} org.opendaylight.mdsal.ServiceEntityType
${SINGLETON_CHANGE_OWNERSHIP_ENTITY_TYPE} org.opendaylight.mdsal.AsyncServiceCloseEntityType
+${NODE_START_COMMAND} ${KARAF_HOME}/bin/start
+${NODE_STOP_COMMAND} ${KARAF_HOME}/bin/stop
+${NODE_KILL_COMMAND} ps axf | grep org.apache.karaf | grep -v grep | awk '{print \"kill -9 \" $1}' | sh
+${NODE_FREEZE_COMMAND} ps axf | grep org.apache.karaf | grep -v grep | awk '{print \"kill -STOP \" $1}' | sh
+${NODE_UNFREEZE_COMMAND} ps axf | grep org.apache.karaf | grep -v grep | awk '{print \"kill -CONT \" $1}' | sh
*** Keywords ***
ClusterManagement_Setup
Check_Cluster_Is_In_Sync
[Arguments] ${member_index_list}=${EMPTY}
[Documentation] Fail if no-sync is detected on a member from list (or any).
- ${index_list} = ClusterManagement__Given_Or_Internal_Index_List given_list=${member_index_list}
+ ${index_list} = List_Indices_Or_All given_list=${member_index_list}
: FOR ${index} IN @{index_list} # usually: 1, 2, 3.
\ ${status} = Get_Sync_Status_Of_Member member_index=${index}
\ BuiltIn.Continue_For_Loop_If 'True' == '${status}'
... If \${validate}, Fail if raft state is not Leader or Follower (for example on Candidate).
... The biggest difference from Get_Leader_And_Followers_For_Shard
... is that no check on number of Leaders is performed.
- ${index_list} = ClusterManagement__Given_Or_Internal_Index_List given_list=${member_index_list}
+ ${index_list} = List_Indices_Or_All given_list=${member_index_list}
Collections.Sort_List ${index_list} # to guarantee return values are also sorted lists
# TODO: Support alternative capitalization of 'config'?
${ds_type} = BuiltIn.Set_Variable_If '${shard_type}' != 'config' operational config
${raft_state} = Collections.Get_From_Dictionary ${value} RaftState
[Return] ${raft_state}
+Verify_Shard_Leader_Elected
+ [Arguments] ${shard_name} ${shard_type} ${new_elected} ${old_leader} ${member_index_list}=${EMPTY}
+ [Documentation] Verify new leader was elected or remained the same. Bool paramter ${new_elected} indicates if
+ ... new leader is elected or should remained the same as ${old_leader}
+ ${leader} ${followers}= Get_Leader_And_Followers_For_Shard shard_name=${shard_name} shard_type=${shard_type} member_index_list=${member_index_list}
+ BuiltIn.Run_Keyword_If ${new_elected} BuiltIn.Should_Not_Be_Equal_As_Numbers ${old_leader} ${leader}
+ BuiltIn.Run_Keyword_Unless ${new_elected} BuiltIn.Should_Be_Equal_As_numbers ${old_leader} ${leader}
+ BuiltIn.Return_From_Keyword ${leader} ${followers}
+
Verify_Owner_And_Successors_For_Device
- [Arguments] ${device_name} ${device_type} ${member_index} ${candidate_list}=${EMPTY}
+ [Arguments] ${device_name} ${device_type} ${member_index} ${candidate_list}=${EMPTY} ${after_stop}=False
[Documentation] Returns the owner and successors for the SB device ${device_name} of type ${device_type}. Request is sent to member ${member_index}.
... For Boron and beyond, candidates are not removed on node down or isolation,
... so this keyword expects candidates to be all members from Boron on.
... ${candidate_list} minus owner is returned as ${successor list}.
... Users can still use Get_Owner_And_Successors_For_Device if they are interested in downed candidates,
... or for testing heterogeneous clusters.
- ${index_list} = ClusterManagement__Given_Or_Internal_Index_List given_list=${candidate_list}
+ ${index_list} = List_Indices_Or_All given_list=${candidate_list}
${owner} ${successor_list} = Get_Owner_And_Successors_For_Device device_name=${device_name} device_type=${device_type} member_index=${member_index}
Collections.List_Should_Contain_Value ${index_list} ${owner} Owner ${owner} is not in candidate list ${index_list}
- ${expected_candidate_list_origin} = CompareStream.Set_Variable_If_At_Least_Boron ${ClusterManagement__member_index_list} ${index_list}
+ # In Beryllium or after stopping an instance, the removed instance does not show in the candidate list.
+ ${expected_candidate_list_origin} = BuiltIn.Set_Variable_If '${ODL_STREAM}' == 'beryllium' or ${after_stop} ${index_list} ${ClusterManagement__member_index_list}
# We do not want to manipulate either origin list.
${expected_successor_list} = BuiltIn.Create_List @{expected_candidate_list_origin}
Collections.Remove_Values_From_List ${expected_successor_list} ${owner}
[Documentation] If the list is empty, kill all ODL instances. Otherwise, kill members based on \${kill_index_list}
... If \${confirm} is True, sleep 1 second and verify killed instances are not there anymore.
... The KW will return a list of available members: \${updated index_list}=\${original_index_list}-\${member_index_list}
- ${kill_index_list} = ClusterManagement__Given_Or_Internal_Index_List given_list=${member_index_list}
- ${index_list} = ClusterManagement__Given_Or_Internal_Index_List given_list=${original_index_list}
- ${command} = BuiltIn.Set_Variable ps axf | grep karaf | grep -v grep | awk '{print \"kill -9 \" $1}' | sh
- Run_Bash_Command_On_List_Or_All command=${command} member_index_list=${member_index_list}
+ ${kill_index_list} = List_Indices_Or_All given_list=${member_index_list}
+ ${index_list} = List_Indices_Or_All given_list=${original_index_list}
+ Run_Bash_Command_On_List_Or_All command=${NODE_KILL_COMMAND} member_index_list=${member_index_list}
${updated_index_list} = BuiltIn.Create_List @{index_list}
Collections.Remove_Values_From_List ${updated_index_list} @{kill_index_list}
BuiltIn.Return_From_Keyword_If not ${confirm} ${updated_index_list}
\ Verify_Karaf_Is_Not_Running_On_Member member_index=${index}
[Return] ${updated_index_list}
+Stop_Single_Member
+ [Arguments] ${member} ${original_index_list}=${EMPTY} ${confirm}=True
+ [Documentation] Convenience keyword that stops the specified member of the cluster.
+ ... The KW will return a list of available members: \${updated index_list}=\${original_index_list}-\${member}
+ ${index_list} = ClusterManagement__Build_List ${member}
+ ${updated_index_list} = Stop_Members_From_List_Or_All ${index_list} ${original_index_list} ${confirm}
+ [Return] ${updated_index_list}
+
+Stop_Members_From_List_Or_All
+ [Arguments] ${member_index_list}=${EMPTY} ${original_index_list}=${EMPTY} ${confirm}=True ${timeout}=120s
+ [Documentation] If the list is empty, stops all ODL instances. Otherwise stop members based on \${stop_index_list}
+ ... If \${confirm} is True, verify stopped instances are not there anymore.
+ ... The KW will return a list of available members: \${updated index_list}=\${original_index_list}-\${member_index_list}
+ ${stop_index_list} = List_Indices_Or_All given_list=${member_index_list}
+ ${index_list} = List_Indices_Or_All given_list=${original_index_list}
+ Run_Bash_Command_On_List_Or_All command=${NODE_STOP_COMMAND} member_index_list=${member_index_list}
+ ${updated_index_list} = BuiltIn.Create_List @{index_list}
+ Collections.Remove_Values_From_List ${updated_index_list} @{stop_index_list}
+ BuiltIn.Return_From_Keyword_If not ${confirm} ${updated_index_list}
+ : FOR ${index} IN @{stop_index_list}
+ \ BuiltIn.Wait Until Keyword Succeeds ${timeout} 2s Verify_Karaf_Is_Not_Running_On_Member member_index=${index}
+ [Return] ${updated_index_list}
+
Start_Single_Member
[Arguments] ${member} ${wait_for_sync}=True ${timeout}=300s
[Documentation] Convenience keyword that starts the specified member of the cluster.
Start_Members_From_List_Or_All ${index_list} ${wait_for_sync} ${timeout}
Start_Members_From_List_Or_All
- [Arguments] ${member_index_list}=${EMPTY} ${wait_for_sync}=True ${timeout}=300s ${karaf_home}=${KARAF_HOME} ${export_java_home}=${JAVA_HOME}
+ [Arguments] ${member_index_list}=${EMPTY} ${wait_for_sync}=True ${timeout}=300s ${karaf_home}=${EMPTY} ${export_java_home}=${EMPTY}
[Documentation] If the list is empty, start all cluster members. Otherwise, start members based on present indices.
... If ${wait_for_sync}, wait for cluster sync on listed members.
... Optionally karaf_home can be overriden. Optionally specific JAVA_HOME is used for starting.
- ${base_command} = BuiltIn.Set_Variable ${karaf_home}/bin/start
+ ${base_command} = BuiltIn.Set_Variable_If "${karaf_home}" ${karaf_home}/bin/start ${NODE_START_COMMAND}
${command} = BuiltIn.Set_Variable_If "${export_java_home}" export JAVA_HOME="${export_java_home}"; ${base_command} ${base_command}
Run_Bash_Command_On_List_Or_All command=${command} member_index_list=${member_index_list}
BuiltIn.Return_From_Keyword_If not ${wait_for_sync}
BuiltIn.Wait_Until_Keyword_Succeeds ${timeout} 10s Check_Cluster_Is_In_Sync member_index_list=${member_index_list}
# TODO: Do we also want to check Shard Leaders here?
+Freeze_Single_Member
+ [Arguments] ${member}
+ [Documentation] Convenience keyword that stops the specified member of the cluster by freezing the jvm.
+ ${index_list} = ClusterManagement__Build_List ${member}
+ Freeze_Or_Unfreeze_Members_From_List_Or_All ${NODE_FREEZE_COMMAND} ${index_list}
+
+Unfreeze_Single_Member
+ [Arguments] ${member} ${wait_for_sync}=True ${timeout}=60s
+ [Documentation] Convenience keyword that "continues" the specified member of the cluster by unfreezing the jvm.
+ ${index_list} = ClusterManagement__Build_List ${member}
+ Freeze_Or_Unfreeze_Members_From_List_Or_All ${NODE_UNFREEZE_COMMAND} ${index_list}
+ BuiltIn.Wait_Until_Keyword_Succeeds ${timeout} 10s Check_Cluster_Is_In_Sync
+
+Freeze_Or_Unfreeze_Members_From_List_Or_All
+ [Arguments] ${command} ${member_index_list}=${EMPTY}
+ [Documentation] If the list is empty, stops/runs all ODL instances. Otherwise stop/run members based on \${stop_index_list}
+ ... For command parameter only ${NODE_FREEZE_COMMAND} and ${NODE_UNFREEZE_COMMAND} should be used
+ ${freeze_index_list} = List_Indices_Or_All given_list=${member_index_list}
+ Run_Bash_Command_On_List_Or_All command=${command} member_index_list=${member_index_list}
+
Clean_Journals_And_Snapshots_On_List_Or_All
[Arguments] ${member_index_list}=${EMPTY} ${karaf_home}=${KARAF_HOME}
[Documentation] Delete journal and snapshots directories on every node listed (or all).
- ${index_list} = ClusterManagement__Given_Or_Internal_Index_List given_list=${member_index_list}
+ ${index_list} = List_Indices_Or_All given_list=${member_index_list}
${command} = Set Variable rm -rf "${karaf_home}/journal" "${karaf_home}/snapshots"
: FOR ${index} IN @{index_list} # usually: 1, 2, 3.
\ Run_Bash_Command_On_Member command=${command} member_index=${index}
Count_Running_Karafs_On_Member
[Arguments] ${member_index}
[Documentation] Remotely execute grep for karaf process, return count as string.
- ${command} = BuiltIn.Set_Variable ps axf | grep karaf | grep -v grep | wc -l
+ ${command} = BuiltIn.Set_Variable ps axf | grep org.apache.karaf | grep -v grep | wc -l
${count} = Run_Bash_Command_On_Member command=${command} member_index=${member_index}
[Return] ${count}
[Arguments] ${isolate_member_index} ${member_index_list}=${EMPTY}
[Documentation] If the list is empty, isolate member from all ODL instances. Otherwise, isolate member based on present indices.
... The KW will return a list of available members: \${updated index_list}=\${member_index_list}-\${isolate_member_index}
- ${index_list} = ClusterManagement__Given_Or_Internal_Index_List given_list=${member_index_list}
+ ${index_list} = List_Indices_Or_All given_list=${member_index_list}
${source} = Collections.Get_From_Dictionary ${ClusterManagement__index_to_ip_mapping} ${isolate_member_index}
: FOR ${index} IN @{index_list}
\ ${destination} = Collections.Get_From_Dictionary ${ClusterManagement__index_to_ip_mapping} ${index}
Rejoin_Member_From_List_Or_All
[Arguments] ${rejoin_member_index} ${member_index_list}=${EMPTY}
[Documentation] If the list is empty, rejoin member from all ODL instances. Otherwise, rejoin member based on present indices.
- ${index_list} = ClusterManagement__Given_Or_Internal_Index_List given_list=${member_index_list}
+ ${index_list} = List_Indices_Or_All given_list=${member_index_list}
${source} = Collections.Get_From_Dictionary ${ClusterManagement__index_to_ip_mapping} ${rejoin_member_index}
: FOR ${index} IN @{index_list}
\ ${destination} = Collections.Get_From_Dictionary ${ClusterManagement__index_to_ip_mapping} ${index}
Run_Bash_Command_On_List_Or_All
[Arguments] ${command} ${member_index_list}=${EMPTY}
[Documentation] Cycle through indices (or all), run command on each.
- ${index_list} = ClusterManagement__Given_Or_Internal_Index_List given_list=${member_index_list}
+ ${index_list} = List_Indices_Or_All given_list=${member_index_list}
: FOR ${index} IN @{index_list}
\ Run_Bash_Command_On_Member command=${command} member_index=${index}
Run_Karaf_Command_On_List_Or_All
[Arguments] ${command} ${member_index_list}=${EMPTY} ${timeout}=10s
[Documentation] Cycle through indices (or all), run karaf command on each.
- ${index_list} = ClusterManagement__Given_Or_Internal_Index_List given_list=${member_index_list}
+ ${index_list} = List_Indices_Or_All given_list=${member_index_list}
: FOR ${index} IN @{index_list}
\ ${member_ip} = Collections.Get_From_Dictionary dictionary=${ClusterManagement__index_to_ip_mapping} key=${index}
\ KarafKeywords.Safe_Issue_Command_On_Karaf_Console ${command} ${member_ip} timeout=${timeout}
Install_Feature_On_List_Or_All
[Arguments] ${feature_name} ${member_index_list}=${EMPTY} ${timeout}=60s
[Documentation] Attempt installation on each member from list (or all). Then look for failures.
- ${index_list} = ClusterManagement__Given_Or_Internal_Index_List given_list=${member_index_list}
+ ${index_list} = List_Indices_Or_All given_list=${member_index_list}
${status_list} = BuiltIn.Create_List
: FOR ${index} IN @{index_list}
\ ${status} ${text} = BuiltIn.Run_Keyword_And_Ignore_Error Install_Feature_On_Member feature_name=${feature_name} member_index=${index}
... Beware that in order to avoid "got positional argument after named arguments", first two arguments in the call should not be named.
BuiltIn.Comment This keyword is experimental and there is high risk of being replaced by another approach.
# TODO: For_Index_From_List_Or_All_Run_Keyword applied to With_Ssh_To_Member_Run_Keyword?
- ${index_list} = ClusterManagement__Given_Or_Internal_Index_List given_list=${member_index_list}
+ ${index_list} = List_Indices_Or_All given_list=${member_index_list}
: FOR ${member_index} IN @{index_list}
\ ${member_ip} = Resolve_IP_Address_For_Member ${member_index}
\ SSHKeywords.Open_Connection_To_Odl_System ip_address=${member_ip}
[Arguments] ${uri} ${expected_data} ${member_index_list}=${EMPTY}
[Documentation] Send a GET with the supplied uri to all or some members defined in ${member_index_list}.
... Then check received data is = ${expected data}.
- ${index_list} = ClusterManagement__Given_Or_Internal_Index_List given_list=${member_index_list}
+ ${index_list} = List_Indices_Or_All given_list=${member_index_list}
: FOR ${index} IN @{index_list}
\ ${data} = Get_From_Member uri=${uri} member_index=${index}
\ TemplatedRequests.Normalize_Jsons_And_Compare ${expected_data} ${data}
[Arguments] ${uri} ${dictionary} ${member_index_list}=${EMPTY}
[Documentation] Send a GET with the supplied uri to all or some members defined in ${member_index_list}.
... Then check received for occurrences of items expressed in a dictionary ${dictionary}.
- ${index_list} = ClusterManagement__Given_Or_Internal_Index_List given_list=${member_index_list}
+ ${index_list} = List_Indices_Or_All given_list=${member_index_list}
: FOR ${index} IN @{index_list}
\ ${data} = Get_From_Member uri=${uri} member_index=${index}
\ Utils.Check Item Occurrence ${data} ${dictionary}
[Arguments] ${uri} ${member_index_list}=${EMPTY}
[Documentation] Send a GET with the supplied uri to all or some members defined in ${member_index_list}.
... Then check there is no content.
- ${index_list} = ClusterManagement__Given_Or_Internal_Index_List given_list=${member_index_list}
+ ${index_list} = List_Indices_Or_All given_list=${member_index_list}
: FOR ${index} IN @{index_list}
\ ${session} = Resolve_Http_Session_For_Member member_index=${index}
\ Utils.No_Content_From_URI ${session} ${uri}
${sync_status} = Collections.Get_From_Dictionary dictionary=${value_object} key=SyncStatus
[Return] ${sync_status}
-List_Indices_Minus_Member
- [Arguments] ${member_index} ${member_index_list}=${EMPTY}
- [Documentation] Create a new list which contains indices from ${member_index_list} (or all) without ${member_index}.
- ${index_list} = ClusterManagement__Given_Or_Empty_List ${member_index_list}
- Collections.Remove Values From List ${index_list} ${member_index}
- [Return] ${index_list}
+List_All_Indices
+ [Documentation] Create a new list of all indices.
+ BuiltIn.Return_Keyword_And_Return List_Indices_Or_All
-ClusterManagement__Given_Or_Internal_Index_List
+List_Indices_Or_All
[Arguments] ${given_list}=${EMPTY}
[Documentation] Utility to allow \${EMPTY} as default argument value, as the internal list is computed at runtime.
- ... This keyword always return a (shallow) copy of given or default list,
+ ... This keyword always returns a (shallow) copy of given or default list,
... so operations with the returned list should not affect other lists.
... Also note that this keyword does not consider empty list to be \${EMPTY}.
- ... TODO: This keyword is frequently used for obtaining copy of ${ClusterManagement__member_index_list}. Give this keyword public name.
${return_list_reference} = BuiltIn.Set_Variable_If """${given_list}""" != "" ${given_list} ${ClusterManagement__member_index_list}
${return_list_copy} = BuiltIn.Create_List @{return_list_reference}
[Return] ${return_list_copy}
-ClusterManagement__Given_Or_Empty_List
- [Arguments] ${given_list}=${EMPTY}
- [Documentation] Utility to allow \${EMPTY} as default argument value, as an empty list is computed at runtime.
- ${empty_list} = BuiltIn.Create_List
- ${given_length} = BuiltIn.Get_Length ${given_list}
- ${return_list} = BuiltIn.Set_Variable_If ${given_length} > 0 ${given_list} ${empty_list}
- [Return] ${return_list}
+List_Indices_Minus_Member
+ [Arguments] ${member_index} ${member_index_list}=${EMPTY}
+ [Documentation] Create a new list which contains indices from ${member_index_list} (or all) without ${member_index}.
+ ${index_list} = List_Indices_Or_All ${member_index_list}
+ Collections.Remove Values From List ${index_list} ${member_index}
+ [Return] ${index_list}
ClusterManagement__Compute_Derived_Variables
[Arguments] ${int_of_members}
${session_alias} = Resolve_Http_Session_For_Member member_index=${index}
RequestsLibrary.Create_Session ${session_alias} http://${member_ip}:${RESTCONFPORT} auth=${AUTH} max_retries=0
Collections.Append_To_List ${session_list} ${session_alias}
+
+Sync_Status_Should_Be_False
+ [Arguments] ${controller_index}
+ [Documentation] Verify that cluster node is not in sync with others
+ ${status} Get_Sync_Status_Of_Member ${controller_index}
+ BuiltIn.Should_Not_Be_True ${status}
+
+Sync_Status_Should_Be_True
+ [Arguments] ${controller_index}
+ [Documentation] Verify that cluster node is in sync with others
+ ${status} Get_Sync_Status_Of_Member ${controller_index}
+ BuiltIn.Should_Be_True ${status}
Code Review / integration / test.git / blobdiff