... - Cluster Setup
... - Shard state, leader and followers
... - Entity Owner, candidates and successors
-... - Kill and Start Member
+... - Kill, Stop and Start Member
... - Isolate and Rejoin Member
... - Run Commands On Member
... - REST requests and checks on Members
${SINGLETON_NETCONF_DEVICE_ID_SUFFIX} ]]]]}']
${SINGLETON_ELECTION_ENTITY_TYPE} org.opendaylight.mdsal.ServiceEntityType
${SINGLETON_CHANGE_OWNERSHIP_ENTITY_TYPE} org.opendaylight.mdsal.AsyncServiceCloseEntityType
+${NODE_START_COMMAND} ${KARAF_HOME}/bin/start
+${NODE_STOP_COMMAND} ${KARAF_HOME}/bin/stop
+${NODE_KILL_COMMAND} ps axf | grep org.apache.karaf | grep -v grep | awk '{print \"kill -9 \" $1}' | sh
+${NODE_FREEZE_COMMAND} ps axf | grep org.apache.karaf | grep -v grep | awk '{print \"kill -STOP \" $1}' | sh
+${NODE_UNFREEZE_COMMAND} ps axf | grep org.apache.karaf | grep -v grep | awk '{print \"kill -CONT \" $1}' | sh
*** Keywords ***
ClusterManagement_Setup
${raft_state} = Collections.Get_From_Dictionary ${value} RaftState
[Return] ${raft_state}
+Verify_Shard_Leader_Elected
+ [Arguments] ${shard_name} ${shard_type} ${new_elected} ${old_leader} ${member_index_list}=${EMPTY}
+ [Documentation] Verify new leader was elected or remained the same. Bool paramter ${new_elected} indicates if
+ ... new leader is elected or should remained the same as ${old_leader}
+ ${leader} ${followers}= Get_Leader_And_Followers_For_Shard shard_name=${shard_name} shard_type=${shard_type} member_index_list=${member_index_list}
+ BuiltIn.Run_Keyword_If ${new_elected} BuiltIn.Should_Not_Be_Equal_As_Numbers ${old_leader} ${leader}
+ BuiltIn.Run_Keyword_Unless ${new_elected} BuiltIn.Should_Be_Equal_As_numbers ${old_leader} ${leader}
+ BuiltIn.Return_From_Keyword ${leader} ${followers}
+
Verify_Owner_And_Successors_For_Device
- [Arguments] ${device_name} ${device_type} ${member_index} ${candidate_list}=${EMPTY}
+ [Arguments] ${device_name} ${device_type} ${member_index} ${candidate_list}=${EMPTY} ${after_stop}=False
[Documentation] Returns the owner and successors for the SB device ${device_name} of type ${device_type}. Request is sent to member ${member_index}.
... For Boron and beyond, candidates are not removed on node down or isolation,
... so this keyword expects candidates to be all members from Boron on.
${index_list} = ClusterManagement__Given_Or_Internal_Index_List given_list=${candidate_list}
${owner} ${successor_list} = Get_Owner_And_Successors_For_Device device_name=${device_name} device_type=${device_type} member_index=${member_index}
Collections.List_Should_Contain_Value ${index_list} ${owner} Owner ${owner} is not in candidate list ${index_list}
- ${expected_candidate_list_origin} = CompareStream.Set_Variable_If_At_Least_Boron ${ClusterManagement__member_index_list} ${index_list}
+ # In Beryllium or after stopping an instance, the removed instance does not show in the candidate list.
+ ${expected_candidate_list_origin} = BuiltIn.Set_Variable_If '${ODL_STREAM}' == 'beryllium' or ${after_stop} ${index_list} ${ClusterManagement__member_index_list}
# We do not want to manipulate either origin list.
${expected_successor_list} = BuiltIn.Create_List @{expected_candidate_list_origin}
Collections.Remove_Values_From_List ${expected_successor_list} ${owner}
... The KW will return a list of available members: \${updated index_list}=\${original_index_list}-\${member_index_list}
${kill_index_list} = ClusterManagement__Given_Or_Internal_Index_List given_list=${member_index_list}
${index_list} = ClusterManagement__Given_Or_Internal_Index_List given_list=${original_index_list}
- ${command} = BuiltIn.Set_Variable ps axf | grep karaf | grep -v grep | awk '{print \"kill -9 \" $1}' | sh
- Run_Bash_Command_On_List_Or_All command=${command} member_index_list=${member_index_list}
+ Run_Bash_Command_On_List_Or_All command=${NODE_KILL_COMMAND} member_index_list=${member_index_list}
${updated_index_list} = BuiltIn.Create_List @{index_list}
Collections.Remove_Values_From_List ${updated_index_list} @{kill_index_list}
BuiltIn.Return_From_Keyword_If not ${confirm} ${updated_index_list}
\ Verify_Karaf_Is_Not_Running_On_Member member_index=${index}
[Return] ${updated_index_list}
+Stop_Single_Member
+ [Arguments] ${member} ${original_index_list}=${EMPTY} ${confirm}=True
+ [Documentation] Convenience keyword that stops the specified member of the cluster.
+ ... The KW will return a list of available members: \${updated index_list}=\${original_index_list}-\${member}
+ ${index_list} = ClusterManagement__Build_List ${member}
+ ${updated_index_list} = Stop_Members_From_List_Or_All ${index_list} ${original_index_list} ${confirm}
+ [Return] ${updated_index_list}
+
+Stop_Members_From_List_Or_All
+ [Arguments] ${member_index_list}=${EMPTY} ${original_index_list}=${EMPTY} ${confirm}=True ${timeout}=120s
+ [Documentation] If the list is empty, stops all ODL instances. Otherwise stop members based on \${stop_index_list}
+ ... If \${confirm} is True, verify stopped instances are not there anymore.
+ ... The KW will return a list of available members: \${updated index_list}=\${original_index_list}-\${member_index_list}
+ ${stop_index_list} = ClusterManagement__Given_Or_Internal_Index_List given_list=${member_index_list}
+ ${index_list} = ClusterManagement__Given_Or_Internal_Index_List given_list=${original_index_list}
+ Run_Bash_Command_On_List_Or_All command=${NODE_STOP_COMMAND} member_index_list=${member_index_list}
+ ${updated_index_list} = BuiltIn.Create_List @{index_list}
+ Collections.Remove_Values_From_List ${updated_index_list} @{stop_index_list}
+ BuiltIn.Return_From_Keyword_If not ${confirm} ${updated_index_list}
+ : FOR ${index} IN @{stop_index_list}
+ \ BuiltIn.Wait Until Keyword Succeeds ${timeout} 2s Verify_Karaf_Is_Not_Running_On_Member member_index=${index}
+ [Return] ${updated_index_list}
+
Start_Single_Member
[Arguments] ${member} ${wait_for_sync}=True ${timeout}=300s
[Documentation] Convenience keyword that starts the specified member of the cluster.
Start_Members_From_List_Or_All ${index_list} ${wait_for_sync} ${timeout}
Start_Members_From_List_Or_All
- [Arguments] ${member_index_list}=${EMPTY} ${wait_for_sync}=True ${timeout}=300s ${karaf_home}=${KARAF_HOME} ${export_java_home}=${JAVA_HOME}
+ [Arguments] ${member_index_list}=${EMPTY} ${wait_for_sync}=True ${timeout}=300s ${karaf_home}=${EMPTY} ${export_java_home}=${EMPTY}
[Documentation] If the list is empty, start all cluster members. Otherwise, start members based on present indices.
... If ${wait_for_sync}, wait for cluster sync on listed members.
... Optionally karaf_home can be overriden. Optionally specific JAVA_HOME is used for starting.
- ${base_command} = BuiltIn.Set_Variable ${karaf_home}/bin/start
+ ${base_command} = BuiltIn.Set_Variable_If "${karaf_home}" ${karaf_home}/bin/start ${NODE_START_COMMAND}
${command} = BuiltIn.Set_Variable_If "${export_java_home}" export JAVA_HOME="${export_java_home}"; ${base_command} ${base_command}
Run_Bash_Command_On_List_Or_All command=${command} member_index_list=${member_index_list}
BuiltIn.Return_From_Keyword_If not ${wait_for_sync}
BuiltIn.Wait_Until_Keyword_Succeeds ${timeout} 10s Check_Cluster_Is_In_Sync member_index_list=${member_index_list}
# TODO: Do we also want to check Shard Leaders here?
+Freeze_Single_Member
+ [Arguments] ${member}
+ [Documentation] Convenience keyword that stops the specified member of the cluster by freezing the jvm.
+ ${index_list} = ClusterManagement__Build_List ${member}
+ Freeze_Or_Unfreeze_Members_From_List_Or_All ${NODE_FREEZE_COMMAND} ${index_list}
+
+Unfreeze_Single_Member
+ [Arguments] ${member} ${wait_for_sync}=True ${timeout}=60s
+ [Documentation] Convenience keyword that "continues" the specified member of the cluster by unfreezing the jvm.
+ ${index_list} = ClusterManagement__Build_List ${member}
+ Freeze_Or_Unfreeze_Members_From_List_Or_All ${NODE_UNFREEZE_COMMAND} ${index_list}
+ BuiltIn.Wait_Until_Keyword_Succeeds ${timeout} 10s Check_Cluster_Is_In_Sync
+
+Freeze_Or_Unfreeze_Members_From_List_Or_All
+ [Arguments] ${command} ${member_index_list}=${EMPTY}
+ [Documentation] If the list is empty, stops/runs all ODL instances. Otherwise stop/run members based on \${stop_index_list}
+ ... For command parameter only ${NODE_FREEZE_COMMAND} and ${NODE_UNFREEZE_COMMAND} should be used
+ ${freeze_index_list} = ClusterManagement__Given_Or_Internal_Index_List given_list=${member_index_list}
+ Run_Bash_Command_On_List_Or_All command=${command} member_index_list=${member_index_list}
+
Clean_Journals_And_Snapshots_On_List_Or_All
[Arguments] ${member_index_list}=${EMPTY} ${karaf_home}=${KARAF_HOME}
[Documentation] Delete journal and snapshots directories on every node listed (or all).
Count_Running_Karafs_On_Member
[Arguments] ${member_index}
[Documentation] Remotely execute grep for karaf process, return count as string.
- ${command} = BuiltIn.Set_Variable ps axf | grep karaf | grep -v grep | wc -l
+ ${command} = BuiltIn.Set_Variable ps axf | grep org.apache.karaf | grep -v grep | wc -l
${count} = Run_Bash_Command_On_Member command=${command} member_index=${member_index}
[Return] ${count}
${path_list} = Builtin.Set Variable If "${directory_list}" == "${EMPTY}" ${ODL_DEFAULT_DATA_PATHS} ${directory_list}
Safe_With_Ssh_To_List_Or_All_Run_Keyword ${member_index_list} ClusterManagement__Clean_Directories ${path_list} ${karaf_home}
+Store_Karaf_Log_On_List_Or_All
+ [Arguments] ${member_index_list}=${EMPTY} ${dst_dir}=/tmp ${karaf_home}=${KARAF_HOME}
+ [Documentation] Saves karaf.log to the ${dst_dir} for members in given list or all. Return None.
+ Safe_With_Ssh_To_List_Or_All_Run_Keyword ${member_index_list} SSHKeywords.Execute_Command_Should_Pass cp ${karaf_home}/data/log/karaf.log ${dst_dir}
+
+Restore_Karaf_Log_On_List_Or_All
+ [Arguments] ${member_index_list}=${EMPTY} ${src_dir}=/tmp ${karaf_home}=${KARAF_HOME}
+ [Documentation] Places stored karaf.log to the ${karaf_home}/data/log for members in given list or all. Return None.
+ Safe_With_Ssh_To_List_Or_All_Run_Keyword ${member_index_list} SSHKeywords.Execute_Command_Should_Pass cp ${src_dir}/karaf.log ${karaf_home}/data/log/
+
ClusterManagement__Clean_Directories
[Arguments] ${relative_path_list} ${karaf_home}
[Documentation] For each relative path, remove files with respect to ${karaf_home}. Return None.
${sync_status} = Collections.Get_From_Dictionary dictionary=${value_object} key=SyncStatus
[Return] ${sync_status}
+List_All_Indices
+ [Documentation] Create a new list of all indices.
+ ${return_list_copy} = ClusterManagement__Given_Or_Internal_Index_List
+ BuiltIn.Return_From_Keyword ${return_list_copy}
+
List_Indices_Minus_Member
[Arguments] ${member_index} ${member_index_list}=${EMPTY}
[Documentation] Create a new list which contains indices from ${member_index_list} (or all) without ${member_index}.
- ${index_list} = ClusterManagement__Given_Or_Empty_List ${member_index_list}
+ ${index_list} = ClusterManagement__Given_Or_Internal_Index_List ${member_index_list}
Collections.Remove Values From List ${index_list} ${member_index}
[Return] ${index_list}
${session_alias} = Resolve_Http_Session_For_Member member_index=${index}
RequestsLibrary.Create_Session ${session_alias} http://${member_ip}:${RESTCONFPORT} auth=${AUTH} max_retries=0
Collections.Append_To_List ${session_list} ${session_alias}
+
+Sync_Status_Should_Be_False
+ [Arguments] ${controller_index}
+ [Documentation] Verify that cluster node is not in sync with others
+ ${status} Get_Sync_Status_Of_Member ${controller_index}
+ BuiltIn.Should_Not_Be_True ${status}
+
+Sync_Status_Should_Be_True
+ [Arguments] ${controller_index}
+ [Documentation] Verify that cluster node is in sync with others
+ ${status} Get_Sync_Status_Of_Member ${controller_index}
+ BuiltIn.Should_Be_True ${status}
Code Review / integration / test.git / blobdiff