... The state includes member indexes, IP addresses and Http (RequestsLibrary) sessions.
... Cluster Keywords normally use member index, member list or nothing (all members) as argument.
...
+... All index lists returned should be sorted numerically, fix if not.
+...
... Requirements:
... odl-jolokia is assumed to be installed.
...
... TODO: Unify capitalization of Leaders and Followers.
Library RequestsLibrary # for Create_Session and To_Json
Library Collections
+Resource ${CURDIR}/CompareStream.robot
Resource ${CURDIR}/TemplatedRequests.robot # for Get_As_Json_From_Uri
Resource ${CURDIR}/Utils.robot # for Run_Command_On_Controller
... The biggest difference from Get_Leader_And_Followers_For_Shard
... is that no check on number of Leaders is performed.
${index_list} = ClusterManagement__Given_Or_Internal_Index_List given_list=${member_index_list}
+ Collections.Sort_List ${index_list} # to guarantee return values are also sorted lists
# TODO: Support alternative capitalization of 'config'?
${ds_type} = BuiltIn.Set_Variable_If '${shard_type}' != 'config' operational config
${leader_list} = BuiltIn.Create_List
Verify_Owner_And_Successors_For_Device
[Arguments] ${device_name} ${device_type} ${member_index} ${candidate_list}=${EMPTY}
[Documentation] Returns the owner and successors for the SB device ${device_name} of type ${device_type}. Request is sent to member ${member_index}.
+ ... For Boron and beyond, candidates are not removed on node down or isolation,
+ ... so this keyword expects candidates to be all members from Boron on.
... Extra check is done to verify owner and successors are within the ${candidate_list}. This KW is useful when combined with WUKS.
+ ... ${candidate_list} minus owner is returned as ${successor list}.
+ ... Users can still use Get_Owner_And_Successors_For_Device if they are interested in downed candidates,
+ ... or for testing heterogeneous clusters.
${index_list} = ClusterManagement__Given_Or_Internal_Index_List given_list=${candidate_list}
${owner} ${successor_list} = Get_Owner_And_Successors_For_Device device_name=${device_name} device_type=${device_type} member_index=${member_index}
Collections.List_Should_Contain_Value ${index_list} ${owner} Owner ${owner} is not in candidate list ${index_list}
- ${expected_successor_list} = BuiltIn.Create_List @{index_list}
+ ${expected_candidate_list_origin} = CompareStream.Set_Variable_If_At_Least_Boron ${ClusterManagement__member_index_list} ${index_list}
+ # We do not want to manipulate either origin list.
+ ${expected_successor_list} = BuiltIn.Create_List @{expected_candidate_list_origin}
Collections.Remove_Values_From_List ${expected_successor_list} ${owner}
- Collections.Lists_Should_Be_Equal ${expected_successor_list} ${successor_list} Successor list ${successor_list} is not in candidate list ${index_list}
- [Return] ${owner} ${successor_list}
+ Collections.Lists_Should_Be_Equal ${expected_successor_list} ${successor_list} Successor list ${successor_list} is not the came as expected ${expected_successor_list}
+ # User expects the returned successor list to be the provided candidate list minus the owner.
+ Collections.Remove_Values_From_List ${index_list} ${owner}
+ [Return] ${owner} ${index_list}
-Get_Owner_And_Successors_For_device
+Get_Owner_And_Successors_For_Device
[Arguments] ${device_name} ${device_type} ${member_index}
[Documentation] Returns the owner and a list of successors for the SB device ${device_name} of type ${device_type}. Request is sent to member ${member_index}.
... Successors are those device candidates not elected as owner. The list of successors = (list of candidates) - (owner).
+ ... The returned successor list is sorted numerically.
+ ... Note that "candidate list" definition currently differs between Beryllium and Boron.
+ ... Use Verify_Owner_And_Successors_For_Device if you want the older semantics (inaccessible nodes not present in the list).
${owner} ${candidate_list} = Get_Owner_And_Candidates_For_Device device_name=${device_name} device_type=${device_type} member_index=${member_index}
- ${successor_list} = BuiltIn.Create_List @{candidate_list}
+ ${successor_list} = BuiltIn.Create_List @{candidate_list} # Copy operation is not required, but new variable name requires a line anyway.
Collections.Remove_Values_From_List ${successor_list} ${owner}
[Return] ${owner} ${successor_list}
[Arguments] ${device_name} ${device_type} ${member_index}
[Documentation] Returns the owner and a list of candidates for the SB device ${device_name} of type ${device_type}. Request is sent to member ${member_index}.
... Candidates are all members that register to own a device, so the list of candiates includes the owner.
+ ... The returned candidate list is sorted numerically.
+ ... Note that "candidate list" definition currently differs between Beryllium and Boron.
+ ... It is recommended to use Get_Owner_And_Successors_For_Device instead of this keyword, see documentation there.
${session} = Resolve_Http_Session_For_Member member_index=${member_index}
${data} = TemplatedRequests.Get_As_Json_From_Uri uri=${ENTITY_OWNER_URI} session=${session}
${candidate_list} = BuiltIn.Create_List
${entity_type} = BuiltIn.Set_Variable_If '${device_type}' == 'netconf' netconf-node/${device_name} ${device_type}
${clear_data} = BuiltIn.Run_Keyword_If '${device_type}' == 'openflow' or '${device_type}' == 'netconf' Extract_OpenFlow_Device_Data ${data}
... ELSE IF '${device_type}' == 'ovsdb' Extract_Ovsdb_Device_Data ${data}
+ ... ELSE IF '${device_type}' == 'org.opendaylight.mdsal.ServiceEntityType' Extract_Service_Entity_Type ${data}
... ELSE Fail Not recognized device type: ${device_type}
${json} = RequestsLibrary.To_Json ${clear_data}
${entity_type_list} = Collections.Get_From_Dictionary &{json}[entity-owners] entity-type
\ ${candidate} = String.Replace_String &{entity_candidate}[name] member- ${EMPTY}
\ ${candidate} = BuiltIn.Convert_To_Integer ${candidate}
\ Collections.Append_To_List ${candidate_list} ${candidate}
+ Collections.Sort_List ${candidate_list}
[Return] ${owner} ${candidate_list}
+Extract_Service_Entity_Type
+ [Arguments] ${data}
+ [Documentation] Remove superfluous device data from Entity Owner printout.
+ ${clear_data} = String.Replace_String ${data} /odl-general-entity:entity[odl-general-entity:name='Uri [_value= ${EMPTY}
+ ${clear_data} = String.Replace_String ${clear_data} ]-service-group'] ${EMPTY}
+ Log ${clear_data}
+ [Return] ${clear_data}
+
Extract_OpenFlow_Device_Data
[Arguments] ${data}
[Documentation] Remove superfluous OpenFlow device data from Entity Owner printout.
- ${clear_data} = String.Replace_String ${data} /general-entity:entity[general-entity:name=' ${EMPTY}
+ ${clear_data} = BuiltIn.Run Keyword If '${ODL_STREAM}' != 'beryllium' and '${ODL_OF_PLUGIN}' == 'lithium' String.Replace_String ${data} org.opendaylight.mdsal.ServiceEntityType openflow
+ ... ELSE BuiltIn.Set_Variable ${data}
+ ${clear_data} = String.Replace_String ${clear_data} /odl-general-entity:entity[odl-general-entity:name=' ${EMPTY}
+ ${clear_data} = String.Replace_String ${clear_data} /general-entity:entity[general-entity:name=' ${EMPTY}
${clear_data} = String.Replace_String ${clear_data} '] ${EMPTY}
Log ${clear_data}
[Return] ${clear_data}
[Return] ${clear_data}
Kill_Single_Member
- [Arguments] ${member} ${confirm}=True
+ [Arguments] ${member} ${original_index_list}=${EMPTY} ${confirm}=True
[Documentation] Convenience keyword that kills the specified member of the cluster.
+ ... The KW will return a list of available members: \${updated index_list}=\${original_index_list}-\${member}
${index_list} = ClusterManagement__Build_List ${member}
- Kill_Members_From_List_Or_All ${index_list} ${confirm}
+ ${updated_index_list} = Kill_Members_From_List_Or_All ${index_list} ${original_index_list} ${confirm}
+ [Return] ${updated_index_list}
Kill_Members_From_List_Or_All
- [Arguments] ${member_index_list}=${EMPTY} ${confirm}=True
- [Documentation] If the list is empty, kill all ODL instances. Otherwise, kill members based on present indices.
+ [Arguments] ${member_index_list}=${EMPTY} ${original_index_list}=${EMPTY} ${confirm}=True
+ [Documentation] If the list is empty, kill all ODL instances. Otherwise, kill members based on \${kill_index_list}
... If \${confirm} is True, sleep 1 second and verify killed instances are not there anymore.
+ ... The KW will return a list of available members: \${updated index_list}=\${original_index_list}-\${member_index_list}
+ ${kill_index_list} = ClusterManagement__Given_Or_Internal_Index_List given_list=${member_index_list}
+ ${index_list} = ClusterManagement__Given_Or_Internal_Index_List given_list=${original_index_list}
${command} = BuiltIn.Set_Variable ps axf | grep karaf | grep -v grep | awk '{print \"kill -9 \" $1}' | sh
Run_Command_On_List_Or_All command=${command} member_index_list=${member_index_list}
- BuiltIn.Return_From_Keyword_If not ${confirm}
+ ${updated_index_list} = BuiltIn.Create_List @{index_list}
+ Collections.Remove_Values_From_List ${updated_index_list} @{kill_index_list}
+ BuiltIn.Return_From_Keyword_If not ${confirm} ${updated_index_list}
# TODO: Convert to WUKS with configurable timeout if it turns out 1 second is not enough.
BuiltIn.Sleep 1s Kill -9 closes open files, which may take longer than ssh overhead, but not long enough to warrant WUKS.
- ${index_list} = ClusterManagement__Given_Or_Internal_Index_List given_list=${member_index_list}
- : FOR ${index} IN @{index_list}
+ : FOR ${index} IN @{kill_index_list}
\ Verify_Karaf_Is_Not_Running_On_Member member_index=${index}
+ [Return] ${updated_index_list}
Start_Single_Member
[Arguments] ${member} ${wait_for_sync}=True ${timeout}=300s
Isolate_Member_From_List_Or_All
[Arguments] ${isolate_member_index} ${member_index_list}=${EMPTY}
[Documentation] If the list is empty, isolate member from all ODL instances. Otherwise, isolate member based on present indices.
+ ... The KW will return a list of available members: \${updated index_list}=\${member_index_list}-\${isolate_member_index}
${index_list} = ClusterManagement__Given_Or_Internal_Index_List given_list=${member_index_list}
${source} = Collections.Get_From_Dictionary ${ClusterManagement__index_to_ip_mapping} ${isolate_member_index}
: FOR ${index} IN @{index_list}
${command} = BuiltIn.Set_Variable sudo /sbin/iptables -L -n
${output} = Run_Command_On_Member command=${command} member_index=${isolate_member_index}
BuiltIn.Log ${output}
+ ${updated_index_list} = BuiltIn.Create_List @{index_list}
+ Collections.Remove_Values_From_List ${updated_index_list} ${isolate_member_index}
+ [Return] ${updated_index_list}
Rejoin_Member_From_List_Or_All
[Arguments] ${rejoin_member_index} ${member_index_list}=${EMPTY}
: FOR ${index} IN @{index_list}
\ Run_Command_On_Member command=${command} member_index=${index}
+Run_Karaf_Command_On_List_Or_All
+ [Arguments] ${command} ${member_index_list}=${EMPTY}
+ [Documentation] Cycle through indices (or all), run karaf command on each.
+ ${index_list} = ClusterManagement__Given_Or_Internal_Index_List given_list=${member_index_list}
+ : FOR ${index} IN @{index_list}
+ \ ${member_ip} = Collections.Get_From_Dictionary dictionary=${ClusterManagement__index_to_ip_mapping} key=${index}
+ \ KarafKeywords.Issue Command On Karaf Console ${command} ${member_ip}
+
Run_Command_On_Member
[Arguments] ${command} ${member_index}
[Documentation] Obtain IP, call Utils and return output. This does not preserve active ssh session.
${response_text} = TemplatedRequests.Get_From_Uri uri=${uri} accept=${access} session=${session}
[Return] ${response_text}
+Resolve_IP_Address_For_Member
+ [Arguments] ${member_index}
+ [Documentation] Return node IP address of given index.
+ ${ip_address} = Collections.Get From Dictionary dictionary=${ClusterManagement__index_to_ip_mapping} key=${member_index}
+ [Return] ${ip_address}
+
Resolve_Http_Session_For_Member
[Arguments] ${member_index}
[Documentation] Return RequestsLibrary session alias pointing to node of given index.
ClusterManagement__Given_Or_Internal_Index_List
[Arguments] ${given_list}=${EMPTY}
[Documentation] Utility to allow \${EMPTY} as default argument value, as the internal list is computed at runtime.
+ ... This keyword always return a (shallow) copy of given or default list,
+ ... so operations with the returned list should not affect other lists.
+ ... Also note that this keyword does not consider empty list to be \${EMPTY}.
+ ${return_list_reference} = BuiltIn.Set_Variable_If """${given_list}""" != "" ${given_list} ${ClusterManagement__member_index_list}
+ ${return_list_copy} = BuiltIn.Create_List @{return_list_reference}
+ [Return] ${return_list_copy}
+
+ClusterManagement__Given_Or_Empty_List
+ [Arguments] ${given_list}=${EMPTY}
+ [Documentation] Utility to allow \${EMPTY} as default argument value, as an empty list is computed at runtime.
+ ${empty_list} = BuiltIn.Create_List
${given_length} = BuiltIn.Get_Length ${given_list}
- ${return_list} = BuiltIn.Set_Variable_If ${given_length} > 0 ${given_list} ${ClusterManagement__member_index_list}
+ ${return_list} = BuiltIn.Set_Variable_If ${given_length} > 0 ${given_list} ${empty_list}
[Return] ${return_list}
ClusterManagement__Compute_Derived_Variables