+Get_Raft_State_Of_Shard_At_Member
+ [Documentation] Send request to Jolokia on indexed member, return extracted Raft status.
+ ... Optionally, check restconf works.
+ [Arguments] ${shard_name} ${shard_type} ${member_index} ${verify_restconf}=False ${http_timeout}=${EMPTY}
+ ${raft_state} = Get_Raft_Property_From_Shard_Member
+ ... RaftState
+ ... ${shard_name}
+ ... ${shard_type}
+ ... ${member_index}
+ ... verify_restconf=${verify_restconf}
+ ... http_timeout=${http_timeout}
+ RETURN ${raft_state}
+
+Get_Raft_State_Of_Shard_Of_All_Member_Nodes
+ [Documentation] Get raft state of shard of all member nodes
+ [Arguments] ${shard_name}=default ${shard_type}=config ${member_index_list}=${EMPTY}
+ ${index_list} = List_Indices_Or_All given_list=${member_index_list}
+ Collections.Sort_List ${index_list}
+ FOR ${index} IN @{index_list}
+ ClusterManagement.Get Raft State Of Shard At Member
+ ... shard_name=${shard_name}
+ ... shard_type=${shard_type}
+ ... member_index=${index}
+ END
+
+Get_Raft_Property_From_Shard_Member
+ [Documentation] Send request to Jolokia on indexed member, return extracted Raft property.
+ ... Optionally, check restconf works.
+ [Arguments] ${property} ${shard_name} ${shard_type} ${member_index} ${verify_restconf}=False ${http_timeout}=${EMPTY}
+ ${session} = Resolve_Http_Session_For_Member member_index=${member_index}
+ # TODO: Does the used URI tend to generate large data which floods log.html?
+ IF ${verify_restconf}
+ TemplatedRequests.Get_As_Json_Templated
+ ... session=${session}
+ ... folder=${RESTCONF_MODULES_DIR}
+ ... verify=False
+ ... http_timeout=${http_timeout}
+ END
+ ${type_class} = Resolve_Shard_Type_Class shard_type=${shard_type}
+ ${cluster_index} = Evaluate ${member_index}+${NODE_ROLE_INDEX_START}-1
+ ${uri} = BuiltIn.Set_Variable
+ ... ${JOLOKIA_READ_URI}:Category=Shards,name=member-${cluster_index}-shard-${shard_name}-${shard_type},type=${type_class}
+ ${data_text} = TemplatedRequests.Get_As_Json_From_Uri
+ ... uri=${uri}
+ ... session=${session}
+ ... http_timeout=${http_timeout}
+ ${data_object} = RequestsLibrary.To_Json ${data_text}
+ ${value} = Collections.Get_From_Dictionary ${data_object} value
+ ${raft_property} = Collections.Get_From_Dictionary ${value} ${property}
+ RETURN ${raft_property}
+
+Verify_Shard_Leader_Elected
+ [Documentation] Verify new leader was elected or remained the same. Bool paramter ${new_elected} indicates if
+ ... new leader is elected or should remained the same as ${old_leader}
+ [Arguments] ${shard_name} ${shard_type} ${new_elected} ${old_leader} ${member_index_list}=${EMPTY} ${verify_restconf}=True
+ ${leader} ${followers} = Get_Leader_And_Followers_For_Shard
+ ... shard_name=${shard_name}
+ ... shard_type=${shard_type}
+ ... member_index_list=${member_index_list}
+ ... verify_restconf=${verify_restconf}
+ IF ${new_elected}
+ BuiltIn.Should_Not_Be_Equal_As_Numbers ${old_leader} ${leader}
+ END
+ IF not ${new_elected}
+ BuiltIn.Should_Be_Equal_As_numbers ${old_leader} ${leader}
+ END
+ RETURN ${leader} ${followers}
+
+Verify_Owner_And_Successors_For_Device
+ [Documentation] Returns the owner and successors for the SB device ${device_name} of type ${device_type}. Request is sent to member ${member_index}.
+ ... For Boron and beyond, candidates are not removed on node down or isolation,
+ ... so this keyword expects candidates to be all members from Boron on.
+ ... Extra check is done to verify owner and successors are within the ${candidate_list}. This KW is useful when combined with WUKS.
+ ... ${candidate_list} minus owner is returned as ${successor list}.
+ ... Users can still use Get_Owner_And_Successors_For_Device if they are interested in downed candidates,
+ ... or for testing heterogeneous clusters.
+ [Arguments] ${device_name} ${device_type} ${member_index} ${candidate_list}=${EMPTY} ${after_stop}=False
+ ${index_list} = List_Indices_Or_All given_list=${candidate_list}
+ ${owner} ${successor_list} = Get_Owner_And_Successors_For_Device
+ ... device_name=${device_name}
+ ... device_type=${device_type}
+ ... member_index=${member_index}
+ Collections.List_Should_Contain_Value
+ ... ${index_list}
+ ... ${owner}
+ ... Owner ${owner} is not in candidate list ${index_list}
+ # In Beryllium or after stopping an instance, the removed instance does not show in the candidate list.
+ ${expected_candidate_list_origin} = BuiltIn.Set_Variable_If
+ ... ${after_stop}
+ ... ${index_list}
+ ... ${ClusterManagement__member_index_list}
+ # We do not want to manipulate either origin list.
+ ${expected_successor_list} = BuiltIn.Create_List @{expected_candidate_list_origin}
+ Collections.Remove_Values_From_List ${expected_successor_list} ${owner}
+ Collections.Lists_Should_Be_Equal
+ ... ${expected_successor_list}
+ ... ${successor_list}
+ ... Successor list ${successor_list} is not the came as expected ${expected_successor_list}
+ # User expects the returned successor list to be the provided candidate list minus the owner.
+ Collections.Remove_Values_From_List ${index_list} ${owner}
+ RETURN ${owner} ${index_list}
+
+Get_Owner_And_Successors_For_Device
+ [Documentation] Returns the owner and a list of successors for the SB device ${device_name} of type ${device_type}. Request is sent to member ${member_index}.
+ ... Successors are those device candidates not elected as owner. The list of successors = (list of candidates) - (owner).
+ ... The returned successor list is sorted numerically.
+ ... Note that "candidate list" definition currently differs between Beryllium and Boron.
+ ... Use Verify_Owner_And_Successors_For_Device if you want the older semantics (inaccessible nodes not present in the list).
+ [Arguments] ${device_name} ${device_type} ${member_index} ${http_timeout}=${EMPTY}
+ # TODO: somewhere to introduce ${DEFAULT_RESTCONF_DATASTORE_TIMEOUT}. Value may depend on protocol (ask vs tell) and perhaps stream.
+ ${owner} ${candidate_list} = Get_Owner_And_Candidates_For_Device
+ ... device_name=${device_name}
+ ... device_type=${device_type}
+ ... member_index=${member_index}
+ ... http_timeout=${http_timeout}
+ # Copy operation is not required, but new variable name requires a line anyway.
+ ${successor_list} = BuiltIn.Create_List
+ ... @{candidate_list}
+ Collections.Remove_Values_From_List ${successor_list} ${owner}
+ RETURN ${owner} ${successor_list}
+
+Get_Owner_And_Candidates_For_Device_Rpc
+ [Documentation] Returns the owner and a list of candidates for the SB device ${device_name} of type ${device_type}. Request is sent to member ${member_index}.
+ ... Candidates are all members that register to own a device, so the list of candiates includes the owner.
+ ... The returned candidate list is sorted numerically.
+ ... Note that "candidate list" definition currently differs between Beryllium and Boron.
+ ... It is recommended to use Get_Owner_And_Successors_For_Device instead of this keyword, see documentation there.
+ [Arguments] ${device_name} ${device_type} ${member_index} ${http_timeout}=${EMPTY}
+ BuiltIn.Comment TODO: Can this implementation be changed to call Get_Owner_And_Candidates_For_Type_And_Id?
+ ${index} = BuiltIn.Convert_To_Integer ${member_index}
+ ${ip} = Resolve_IP_Address_For_Member member_index=${index}
+ ${entity_type} = BuiltIn.Set_Variable_If
+ ... '${device_type}' == 'netconf'
+ ... netconf-node/${device_name}
+ ... ${device_type}
+ ${url} = BuiltIn.Catenate SEPARATOR= http:// ${ip} :8181/ ${RESTCONF_URI}
+ ${entity_result} = ClusterEntities.Get_Entity ${url} ${entity_type} ${device_name}
+ ${entity_candidates} = Collections.Get_From_Dictionary ${entity_result} candidates
+ ${entity_owner} = Collections.Get_From_Dictionary ${entity_result} owner
+ BuiltIn.Should_Not_Be_Empty ${entity_owner} No owner found for ${device_name}
+ ${owner} = String.Replace_String ${entity_owner} member- ${EMPTY}
+ ${owner} = BuiltIn.Convert_To_Integer ${owner}
+ ${candidate_list} = BuiltIn.Create_List
+ FOR ${entity_candidate} IN @{entity_candidates}
+ ${candidate} = String.Replace_String ${entity_candidate} member- ${EMPTY}
+ ${candidate} = BuiltIn.Convert_To_Integer ${candidate}
+ Collections.Append_To_List ${candidate_list} ${candidate}
+ END
+ Collections.Sort_List ${candidate_list}
+ RETURN ${owner} ${candidate_list}
+
+Get_Owner_And_Candidates_For_Device_Singleton
+ [Documentation] Returns the owner and a list of candidates for the SB device ${device_name} of type ${device_type}. Request is sent to member ${member_index}.
+ [Arguments] ${device_name} ${device_type} ${member_index} ${http_timeout}=${EMPTY}
+ # Normalize device type to the lowercase as in ${SINGLETON_DEVICE_ID_PREFIX} & ${SINGLETON_DEVICE_ID_SUFFIX}
+ ${device_type} = String.Convert To Lower Case ${device_type}
+ # Set device ID prefix
+ Collections.Dictionary Should Contain Key ${SINGLETON_DEVICE_ID_PREFIX} ${device_type}
+ ${device_id_prefix} = Collections.Get From Dictionary ${SINGLETON_DEVICE_ID_PREFIX} ${device_type}
+ Log ${device_id_prefix}
+ # Set device ID suffix
+ Collections.Dictionary Should Contain Key ${SINGLETON_DEVICE_ID_SUFFIX} ${device_type}
+ ${device_id_suffix} = Collections.Get From Dictionary ${SINGLETON_DEVICE_ID_SUFFIX} ${device_type}
+ Log ${device_id_suffix}
+ # Set device ID
+ ${id} = BuiltIn.Set_Variable ${device_id_prefix}${device_name}${device_id_suffix}
+ Log ${id}
+ # Get election entity type results
+ ${type} = BuiltIn.Set_Variable ${SINGLETON_ELECTION_ENTITY_TYPE}
+ ${owner_1} ${candidate_list_1} = Get_Owner_And_Candidates_For_Type_And_Id
+ ... ${type}
+ ... ${id}
+ ... ${member_index}
+ ... http_timeout=${http_timeout}
+ # Get change ownership entity type results
+ ${type} = BuiltIn.Set_Variable ${SINGLETON_CHANGE_OWNERSHIP_ENTITY_TYPE}
+ ${owner_2} ${candidate_list_2} = Get_Owner_And_Candidates_For_Type_And_Id
+ ... ${type}
+ ... ${id}
+ ... ${member_index}
+ ... http_timeout=${http_timeout}
+ # Owners must be same, if not, there is still some election or change ownership in progress
+ BuiltIn.Should_Be_Equal_As_Integers ${owner_1} ${owner_2} Owners for device ${device_name} are not same
+ RETURN ${owner_1} ${candidate_list_1}
+
+Get_Owner_And_Candidates_For_Device
+ [Documentation] Returns the owner and a list of candidates for the SB device ${device_name} of type ${device_type}. Request is sent to member ${member_index}.
+ ... If parsing as singleton failed, kw try to parse data in old way (without singleton).
+ ... Candidates are all members that register to own a device, so the list of candiates includes the owner.
+ ... The returned candidate list is sorted numerically.
+ ... Note that "candidate list" definition currently differs between Beryllium and Boron.
+ ... It is recommended to use Get_Owner_And_Successors_For_Device instead of this keyword, see documentation there.
+ [Arguments] ${device_name} ${device_type} ${member_index} ${http_timeout}=${EMPTY}
+ # Try singleton
+ ${status} ${results} = BuiltIn.Run_Keyword_And_Ignore_Error
+ ... Get_Owner_And_Candidates_For_Device_Singleton
+ ... device_name=${device_name}
+ ... device_type=${device_type}
+ ... member_index=${member_index}
+ ... http_timeout=${http_timeout}
+ IF "${status}"=="PASS" RETURN ${results}
+ # If singleton failed, try parsing in old way
+ ${status} ${results} = BuiltIn.Run_Keyword_And_Ignore_Error
+ ... Get_Owner_And_Candidates_For_Device_Rpc
+ ... device_name=${device_name}
+ ... device_type=${device_type}
+ ... member_index=${member_index}
+ ... http_timeout=${http_timeout}
+ # previous 3 lines (BuilIn.Return.., # If singleton..., ${status}....) could be deleted when old way will not be supported anymore
+ IF '${status}'=='FAIL'
+ BuiltIn.Fail Could not parse owner and candidates for device ${device_name}
+ END
+ RETURN @{results}
+
+Check_Old_Owner_Stays_Elected_For_Device
+ [Documentation] Verify the owner remain the same as ${old_owner}
+ [Arguments] ${device_name} ${device_type} ${old_owner} ${node_to_ask} ${http_timeout}=${EMPTY}
+ ${owner} ${candidates} = Get_Owner_And_Candidates_For_Device
+ ... ${device_name}
+ ... ${device_type}
+ ... ${node_to_ask}
+ ... http_timeout=${http_timeout}
+ BuiltIn.Should_Be_Equal_As_numbers ${old_owner} ${owner}
+ RETURN ${owner} ${candidates}
+
+Check_New_Owner_Got_Elected_For_Device
+ [Documentation] Verify new owner was elected comparing to ${old_owner}
+ [Arguments] ${device_name} ${device_type} ${old_owner} ${node_to_ask} ${http_timeout}=${EMPTY}
+ ${owner} ${candidates} = Get_Owner_And_Candidates_For_Device
+ ... ${device_name}
+ ... ${device_type}
+ ... ${node_to_ask}
+ ... http_timeout=${http_timeout}
+ BuiltIn.Should_Not_Be_Equal_As_Numbers ${old_owner} ${owner}
+ RETURN ${owner} ${candidates}
+
+Get_Owner_And_Candidates_For_Type_And_Id
+ [Documentation] Returns the owner and a list of candidates for entity specified by ${type} and ${id}
+ ... Request is sent to member ${member_index}.
+ ... Candidates are all members that register to own a device, so the list of candiates includes the owner.
+ ... Bear in mind that for Boron and beyond, candidates are not removed on node down or isolation.
+ ... If ${require_candidate_list} is not \${EMPTY}, check whether the actual list of candidates matches.
+ ... Note that differs from "given list" semantics used in other keywords,
+ ... namely you cannot use \${EMPTY} to stand for "full list" in this keyword.
+ [Arguments] ${type} ${id} ${member_index} ${require_candidate_list}=${EMPTY} ${http_timeout}=${EMPTY}
+ BuiltIn.Comment TODO: Find a way to unify and deduplicate code blocks in Get_Owner_And_Candidates_* keywords.
+ ${owner} ${candidates} = Get_Owner_And_Candidates_For_Device_Rpc
+ ... ${id}
+ ... ${type}
+ ... ${member_index}
+ ... http_timeout=${http_timeout}
+ RETURN ${owner} ${candidates}
+
+Extract_Service_Entity_Type
+ [Documentation] Remove superfluous device data from Entity Owner printout.
+ [Arguments] ${data}
+ ${clear_data} = String.Replace_String
+ ... ${data}
+ ... /odl-general-entity:entity[odl-general-entity:name='
+ ... ${EMPTY}
+ ${clear_data} = String.Replace_String ${clear_data} -service-group'] ${EMPTY}
+ Log ${clear_data}
+ RETURN ${clear_data}
+
+Extract_OpenFlow_Device_Data
+ [Documentation] Remove superfluous OpenFlow device data from Entity Owner printout.
+ [Arguments] ${data}
+ ${clear_data} = String.Replace_String ${data} org.opendaylight.mdsal.ServiceEntityType openflow
+ ${clear_data} = String.Replace_String
+ ... ${clear_data}
+ ... /odl-general-entity:entity[odl-general-entity:name='
+ ... ${EMPTY}
+ ${clear_data} = String.Replace_String
+ ... ${clear_data}
+ ... /general-entity:entity[general-entity:name='
+ ... ${EMPTY}
+ ${clear_data} = String.Replace_String ${clear_data} '] ${EMPTY}
+ Log ${clear_data}
+ RETURN ${clear_data}
+
+Extract_Ovsdb_Device_Data
+ [Documentation] Remove superfluous OVSDB device data from Entity Owner printout.
+ [Arguments] ${data}
+ ${clear_data} = String.Replace_String
+ ... ${data}
+ ... /network-topology:network-topology/network-topology:topology[network-topology:topology-id='ovsdb:1']/network-topology:node[network-topology:node-id='
+ ... ${EMPTY}
+ ${clear_data} = String.Replace_String ${clear_data} '] ${EMPTY}
+ Log ${clear_data}
+ RETURN ${clear_data}
+
+Kill_Single_Member
+ [Documentation] Convenience keyword that kills the specified member of the cluster.
+ ... The KW will return a list of available members: \${updated index_list}=\${original_index_list}-\${member}
+ [Arguments] ${member} ${original_index_list}=${EMPTY} ${confirm}=True
+ ${index_list} = ClusterManagement__Build_List ${member}
+ ${member_ip} = Return_Member_IP ${member}
+ KarafKeywords.Log_Message_To_Controller_Karaf Killing ODL${member} ${member_ip}
+ ${updated_index_list} = Kill_Members_From_List_Or_All ${index_list} ${original_index_list} ${confirm}
+ RETURN ${updated_index_list}
+
+Kill_Members_From_List_Or_All
+ [Documentation] If the list is empty, kill all ODL instances. Otherwise, kill members based on \${kill_index_list}
+ ... If \${confirm} is True, sleep 1 second and verify killed instances are not there anymore.
+ ... The KW will return a list of available members: \${updated index_list}=\${original_index_list}-\${member_index_list}
+ [Arguments] ${member_index_list}=${EMPTY} ${original_index_list}=${EMPTY} ${confirm}=True
+ ${kill_index_list} = List_Indices_Or_All given_list=${member_index_list}
+ ${index_list} = List_Indices_Or_All given_list=${original_index_list}
+ Run_Bash_Command_On_List_Or_All command=${NODE_KILL_COMMAND} member_index_list=${member_index_list}
+ ${updated_index_list} = BuiltIn.Create_List @{index_list}
+ Collections.Remove_Values_From_List ${updated_index_list} @{kill_index_list}
+ IF not ${confirm} RETURN ${updated_index_list}
+ # TODO: Convert to WUKS with configurable timeout if it turns out 1 second is not enough.
+ BuiltIn.Sleep
+ ... 1s
+ ... Kill -9 closes open files, which may take longer than ssh overhead, but not long enough to warrant WUKS.
+ FOR ${index} IN @{kill_index_list}
+ Verify_Karaf_Is_Not_Running_On_Member member_index=${index}
+ END
+ Run_Bash_Command_On_List_Or_All command=netstat -pnatu | grep 2550
+ RETURN ${updated_index_list}
+
+Stop_Single_Member
+ [Documentation] Convenience keyword that stops the specified member of the cluster.
+ ... The KW will return a list of available members: \${updated index_list}=\${original_index_list}-\${member}
+ [Arguments] ${member} ${original_index_list}=${EMPTY} ${confirm}=True ${msg}=${EMPTY}
+ ${index_list} = ClusterManagement__Build_List ${member}
+ ${member_ip} = Return_Member_IP ${member}
+ ${msg} = Builtin.Set Variable If
+ ... "${msg}" == "${EMPTY}"
+ ... Stopping ODL${member} ${member_ip}
+ ... Stopping ODL${member} ${member_ip}, ${msg}
+ KarafKeywords.Log_Message_To_Controller_Karaf ${msg}
+ ${updated_index_list} = Stop_Members_From_List_Or_All ${index_list} ${original_index_list} ${confirm}
+ RETURN ${updated_index_list}
+
+Stop_Members_From_List_Or_All
+ [Documentation] If the list is empty, stops all ODL instances. Otherwise stop members based on \${stop_index_list}
+ ... If \${confirm} is True, verify stopped instances are not there anymore.
+ ... The KW will return a list of available members: \${updated index_list}=\${original_index_list}-\${member_index_list}
+ [Arguments] ${member_index_list}=${EMPTY} ${original_index_list}=${EMPTY} ${confirm}=True ${timeout}=360s
+ ${stop_index_list} = List_Indices_Or_All given_list=${member_index_list}
+ ${index_list} = List_Indices_Or_All given_list=${original_index_list}
+ Run_Bash_Command_On_List_Or_All command=${NODE_STOP_COMMAND} member_index_list=${member_index_list}
+ ${updated_index_list} = BuiltIn.Create_List @{index_list}
+ Collections.Remove_Values_From_List ${updated_index_list} @{stop_index_list}
+ IF not ${confirm} RETURN ${updated_index_list}
+ FOR ${index} IN @{stop_index_list}
+ BuiltIn.Wait Until Keyword Succeeds
+ ... ${timeout}
+ ... 2s
+ ... Verify_Karaf_Is_Not_Running_On_Member
+ ... member_index=${index}
+ END
+ Run_Bash_Command_On_List_Or_All command=netstat -pnatu | grep 2550
+ RETURN ${updated_index_list}
+
+Start_Single_Member
+ [Documentation] Convenience keyword that starts the specified member of the cluster.
+ [Arguments] ${member} ${wait_for_sync}=True ${timeout}=300s ${msg}=${EMPTY} ${check_system_status}=False ${verify_restconf}=True
+ ... ${service_list}=${EMPTY_LIST}
+ ${index_list} = ClusterManagement__Build_List ${member}
+ ${member_ip} = Return_Member_IP ${member}
+ ${msg} = Builtin.Set Variable If
+ ... "${msg}" == "${EMPTY}"
+ ... Starting ODL${member} ${member_ip}
+ ... Starting ODL${member} ${member_ip}, ${msg}
+ KarafKeywords.Log_Message_To_Controller_Karaf ${msg}
+ Start_Members_From_List_Or_All
+ ... ${index_list}
+ ... ${wait_for_sync}
+ ... ${timeout}
+ ... check_system_status=${check_system_status}
+ ... verify_restconf=${verify_restconf}
+ ... service_list=${service_list}
+
+Start_Members_From_List_Or_All
+ [Documentation] If the list is empty, start all cluster members. Otherwise, start members based on present indices.
+ ... If ${wait_for_sync}, wait for cluster sync on listed members.
+ ... Optionally karaf_home can be overriden. Optionally specific JAVA_HOME is used for starting.
+ ... Garbage collection is unconditionally logged to files. TODO: Make that reasonable conditional?
+ [Arguments] ${member_index_list}=${EMPTY} ${wait_for_sync}=True ${timeout}=360s ${karaf_home}=${EMPTY} ${export_java_home}=${EMPTY} ${gc_log_dir}=${EMPTY}
+ ... ${check_system_status}=False ${verify_restconf}=True ${service_list}=${EMPTY_LIST}
+ ${base_command} = BuiltIn.Set_Variable_If
+ ... """${karaf_home}""" != ""
+ ... ${karaf_home}/bin/start
+ ... ${NODE_START_COMMAND}
+ ${command} = BuiltIn.Set_Variable_If
+ ... """${export_java_home}""" != ""
+ ... export JAVA_HOME="${export_java_home}"; ${base_command}
+ ... ${base_command}
+ ${epoch} = DateTime.Get_Current_Date time_zone=UTC result_format=epoch exclude_millis=False
+ ${gc_filepath} = BuiltIn.Set_Variable_If
+ ... """${karaf_home}""" != ""
+ ... ${karaf_home}/data/log/gc_${epoch}.log
+ ... ${GC_LOG_PATH}/gc_${epoch}.log
+ ${gc_options} = BuiltIn.Set_Variable_If
+ ... "docker" not in """${node_start_command}"""
+ ... -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:${gc_filepath}
+ ... ${EMPTY}
+ Run_Bash_Command_On_List_Or_All command=${command} ${gc_options} member_index_list=${member_index_list}
+ BuiltIn.Wait_Until_Keyword_Succeeds
+ ... ${timeout}
+ ... 10s
+ ... Verify_Members_Are_Ready
+ ... ${member_index_list}
+ ... ${wait_for_sync}
+ ... ${verify_restconf}
+ ... ${check_system_status}
+ ... ${service_list}
+ [Teardown] Run_Bash_Command_On_List_Or_All command=netstat -pnatu | grep 2550
+
+Verify_Members_Are_Ready
+ [Documentation] Verifies the specified readiness conditions for the given listed members after startup.
+ ... If ${verify_cluster_sync}, verifies the datastores have synced with the rest of the cluster.
+ ... If ${verify_restconf}, verifies RESTCONF is available.
+ ... If ${verify_system_status}, verifies the system services are OPERATIONAL.
+ [Arguments] ${member_index_list} ${verify_cluster_sync} ${verify_restconf} ${verify_system_status} ${service_list}
+ IF ${verify_cluster_sync}
+ Check_Cluster_Is_In_Sync ${member_index_list}
+ END
+ IF ${verify_restconf}
+ Verify_Restconf_Is_Available ${member_index_list}
+ END
+ # for backward compatibility, some consumers might not be passing @{service_list}, but since we can't set a list to a default
+ # value, we need to check here if it's empty in order to skip the check which would throw an error
+ IF ${verify_system_status} and ("${service_list}" != "[[]]")
+ ClusterManagement.Check Status Of Services Is OPERATIONAL @{service_list}
+ END
+
+Verify_Restconf_Is_Available
+ [Arguments] ${member_index_list}
+ ${index_list} = List_Indices_Or_All given_list=${member_index_list}
+ FOR ${index} IN @{index_list}
+ ${session} = Resolve_Http_Session_For_Member member_index=${index}
+ TemplatedRequests.Get_As_Json_Templated session=${session} folder=${RESTCONF_MODULES_DIR} verify=False
+ END
+
+Freeze_Single_Member
+ [Documentation] Convenience keyword that stops the specified member of the cluster by freezing the jvm.
+ [Arguments] ${member}
+ ${index_list} = ClusterManagement__Build_List ${member}
+ Freeze_Or_Unfreeze_Members_From_List_Or_All ${NODE_FREEZE_COMMAND} ${index_list}
+
+Unfreeze_Single_Member
+ [Documentation] Convenience keyword that "continues" the specified member of the cluster by unfreezing the jvm.
+ [Arguments] ${member} ${wait_for_sync}=True ${timeout}=60s
+ ${index_list} = ClusterManagement__Build_List ${member}
+ Freeze_Or_Unfreeze_Members_From_List_Or_All ${NODE_UNFREEZE_COMMAND} ${index_list}
+ BuiltIn.Wait_Until_Keyword_Succeeds ${timeout} 10s Check_Cluster_Is_In_Sync
+
+Freeze_Or_Unfreeze_Members_From_List_Or_All
+ [Documentation] If the list is empty, stops/runs all ODL instances. Otherwise stop/run members based on \${stop_index_list}
+ ... For command parameter only ${NODE_FREEZE_COMMAND} and ${NODE_UNFREEZE_COMMAND} should be used
+ [Arguments] ${command} ${member_index_list}=${EMPTY}
+ ${freeze_index_list} = List_Indices_Or_All given_list=${member_index_list}
+ Run_Bash_Command_On_List_Or_All command=${command} member_index_list=${member_index_list}
+
+Clean_Journals_Data_And_Snapshots_On_List_Or_All
+ [Documentation] Delete journal and snapshots directories on every node listed (or all).
+ ... BEWARE: If only a subset of members is cleaned, this causes RetiredGenerationException in Carbon after the affected node re-start.
+ ... See https://bugs.opendaylight.org/show_bug.cgi?id=8138
+ [Arguments] ${member_index_list}=${EMPTY} ${karaf_home}=${KARAF_HOME}
+ ${index_list} = List_Indices_Or_All given_list=${member_index_list}
+ ${command} = Set Variable rm -rf "${karaf_home}/"*journal "${karaf_home}/snapshots" "${karaf_home}/data"
+ FOR ${index} IN @{index_list} # usually: 1, 2, 3.
+ Run_Bash_Command_On_Member command=${command} member_index=${index}
+ END