ClusterManagement.Flush_Iptables_From_List_Or_All
: FOR ${i} IN RANGE ${NUM_ODL_SYSTEM}
\ BuiltIn.Wait Until Keyword Succeeds 240 1 ClusterManagement.Sync_Status_Should_Be_True ${i+1}
- ${controller_index} = Get Active Controller
+ ${controller_index} = Get Leader Controller
SxpLib.Delete Node ${DEVICE_NODE_ID} session=${DEVICE_SESSION}
SxpLib.Delete Node ${CLUSTER_NODE_ID} session=controller${controller_index}
${resp} = SxpLib.Get Connections node=${node} session=${session}
SxpLib.Should Contain Connection ${resp} ${TOOLS_SYSTEM_IP} ${port} ${mode} ${version}
-Get Active Controller
- [Documentation] Find cluster controller that is marked as leader for SXP service in cluster
- @{votes} = BuiltIn.Create List
+Get Leader Controller
+ [Documentation] Find cluster controller that is marked as leader in cluster with all members running
+ @{running_members} = BuiltIn.Create List
: FOR ${i} IN RANGE ${NUM_ODL_SYSTEM}
- \ ${resp} = RequestsLibrary.Get Request controller${i+1} /restconf/operational/entity-owners:entity-owners
+ \ Collections.Append To List ${running_members} ${i+1}
+ ${active_controller} = Get Leader Controller From Running @{running_members}
+ [Return] ${active_controller}
+
+Get Leader Controller From Running
+ [Arguments] @{running_members}
+ [Documentation] Find cluster controller that is marked as leader in cluster with only some members running
+ BuiltIn.Log ${running_members}
+ @{votes} = BuiltIn.Create List
+ : FOR ${i} IN @{running_members}
+ \ ${resp} = RequestsLibrary.Get Request controller${i} /restconf/operational/entity-owners:entity-owners
\ BuiltIn.Continue For Loop If ${resp.status_code} != 200
- \ ${controller} = Sxp.Get Active Controller From Json ${resp.content} SxpControllerInstance
+ \ ${controller} = Sxp.Get Leader Controller From Json ${resp.content} SxpControllerInstance
\ Collections.Append To List ${votes} ${controller}
${length} = BuiltIn.Get Length ${votes}
BuiltIn.Should Not Be Equal As Integers ${length} 0
Get Inactive Controller
[Documentation] Find cluster controller that is not marked as leader for SXP service in cluster
- ${active_controller} = Get Active Controller
+ ${active_controller} = Get Leader Controller
${controller} = BuiltIn.Evaluate random.choice( filter( lambda i: i!=${active_controller}, range(1, ${NUM_ODL_SYSTEM} + 1))) random
[Return] ${controller}
Resource ../../../libraries/ClusterManagement.robot
Resource ../../../libraries/SxpClusterLib.robot
-*** Variables ***
-${SAMPLES} 1
-${MAC_ADDRESS_TABLE} &{EMPTY}
-${VIRTUAL_IP_1} ${TOOLS_SYSTEM_2_IP}
-${VIRTUAL_INTERFACE_1} eth0:0
-${VIRTUAL_IP_MASK_1} 255.255.255.0
-
*** Test Cases ***
Route Definition Test
[Documentation] Test Route update mechanism without cluster node isolation
SxpClusterLib.Check Shards Status
- ${active_controller} = SxpClusterLib.Get Active Controller
- BuiltIn.Wait Until Keyword Succeeds 240 1 SxpClusterLib.Ip Addres Should Not Be Routed To Follower ${MAC_ADDRESS_TABLE} ${VIRTUAL_IP_1} ${active_controller}
- Add Route Definition To Cluster ${VIRTUAL_IP_1} ${VIRTUAL_IP_MASK_1} ${VIRTUAL_INTERFACE_1} ${active_controller}
- BuiltIn.Wait Until Keyword Succeeds 240 1 SxpClusterLib.Ip Addres Should Be Routed To Follower ${MAC_ADDRESS_TABLE} ${VIRTUAL_IP_1} ${active_controller}
+ ${active_controller} = SxpClusterLib.Get Leader Controller
+ BuiltIn.Wait Until Keyword Succeeds 240 1 SxpClusterLib.Ip Addres Should Not Be Routed To Follower ${MAC_ADDRESS_TABLE} ${VIRTUAL_IP} ${active_controller}
+ Add Route Definition To Cluster ${VIRTUAL_IP} ${VIRTUAL_IP_MASK} ${VIRTUAL_INTERFACE} ${active_controller}
+ BuiltIn.Wait Until Keyword Succeeds 240 1 SxpClusterLib.Ip Addres Should Be Routed To Follower ${MAC_ADDRESS_TABLE} ${VIRTUAL_IP} ${active_controller}
SxpLib.Clean Routing Configuration To Controller controller${active_controller}
- BuiltIn.Wait Until Keyword Succeeds 240 1 SxpClusterLib.Ip Addres Should Not Be Routed To Follower ${MAC_ADDRESS_TABLE} ${VIRTUAL_IP_1} ${active_controller}
- Put Route Definition To Cluster ${VIRTUAL_IP_1} ${VIRTUAL_IP_MASK_1} ${VIRTUAL_INTERFACE_1} ${active_controller}
- BuiltIn.Wait Until Keyword Succeeds 240 1 SxpClusterLib.Ip Addres Should Be Routed To Follower ${MAC_ADDRESS_TABLE} ${VIRTUAL_IP_1} ${active_controller}
+ BuiltIn.Wait Until Keyword Succeeds 240 1 SxpClusterLib.Ip Addres Should Not Be Routed To Follower ${MAC_ADDRESS_TABLE} ${VIRTUAL_IP} ${active_controller}
+ Put Route Definition To Cluster ${VIRTUAL_IP} ${VIRTUAL_IP_MASK} ${VIRTUAL_INTERFACE} ${active_controller}
+ BuiltIn.Wait Until Keyword Succeeds 240 1 SxpClusterLib.Ip Addres Should Be Routed To Follower ${MAC_ADDRESS_TABLE} ${VIRTUAL_IP} ${active_controller}
Isolation of SXP service follower Test
[Documentation] Test Route update mechanism during Cluster isolation,
... after each isolation virtual ip should be pre-routed to new leader
SxpClusterLib.Check Shards Status
${any_controller} = SxpClusterLib.Get Any Controller
- Add Route Definition To Cluster ${VIRTUAL_IP_1} ${VIRTUAL_IP_MASK_1} ${VIRTUAL_INTERFACE_1} ${any_controller}
- : FOR ${i} IN RANGE 0 ${SAMPLES}
- \ ${controller_index} = SxpClusterLib.Get Active Controller
- \ Isolate SXP Controller ${controller_index}
+ Add Route Definition To Cluster ${VIRTUAL_IP} ${VIRTUAL_IP_MASK} ${VIRTUAL_INTERFACE} ${any_controller}
+ ${controller_index} = SxpClusterLib.Get Leader Controller
+ Isolate SXP Controller ${controller_index}
*** Keywords ***
Put Route Definition To Cluster
Custom Clean SXP Cluster
[Documentation] Cleans up Route definitions
- ${follower} = SxpClusterLib.Get Active Controller
+ ${follower} = SxpClusterLib.Get Leader Controller
SxpLib.Clean Routing Configuration To Controller controller${follower}
Setup Custom SXP Cluster Session
Clean Custom SXP Cluster Session
[Documentation] Cleans up resources generated by test
- ${controller_index} = SxpClusterLib.Get Active Controller
+ ${controller_index} = SxpClusterLib.Get Leader Controller
SxpLib.Clean Routing Configuration To Controller controller${controller_index}
SxpClusterLib.Clean SXP Cluster Session
SxpClusterLib.Delete Virtual Interface
[Arguments] ${controller_index}
[Documentation] Isolate one of cluster nodes and perform check that virtual ip is routed to another cluster node,
... afterwards unisolate old leader.
- ClusterManagement.Isolate_Member_From_List_Or_All ${controller_index}
- BuiltIn.Wait Until Keyword Succeeds 120 1 ClusterManagement.Sync_Status_Should_Be_False ${controller_index}
- BuiltIn.Wait Until Keyword Succeeds 120 1 SxpClusterLib.Ip Addres Should Not Be Routed To Follower ${MAC_ADDRESS_TABLE} ${VIRTUAL_IP_1} ${controller_index}
- ${active_follower} = SxpClusterLib.Get Active Controller
- BuiltIn.Wait Until Keyword Succeeds 120 1 SxpClusterLib.Ip Addres Should Be Routed To Follower ${MAC_ADDRESS_TABLE} ${VIRTUAL_IP_1} ${active_follower}
+ @{running_members} = ClusterManagement.Isolate_Member_From_List_Or_All ${controller_index}
+ BuiltIn.Wait Until Keyword Succeeds 240 1 ClusterManagement.Sync_Status_Should_Be_False ${controller_index}
+ BuiltIn.Wait Until Keyword Succeeds 240 1 SxpClusterLib.Ip Addres Should Not Be Routed To Follower ${MAC_ADDRESS_TABLE} ${VIRTUAL_IP} ${controller_index}
+ ${active_follower} = SxpClusterLib.Get Leader Controller From Running @{running_members}
+ BuiltIn.Wait Until Keyword Succeeds 240 1 SxpClusterLib.Ip Addres Should Be Routed To Follower ${MAC_ADDRESS_TABLE} ${VIRTUAL_IP} ${active_follower}
ClusterManagement.Flush_Iptables_From_List_Or_All
- BuiltIn.Wait Until Keyword Succeeds 120 1 ClusterManagement.Sync_Status_Should_Be_True ${controller_index}
+ BuiltIn.Wait Until Keyword Succeeds 240 1 ClusterManagement.Sync_Status_Should_Be_True ${controller_index}
[Documentation] Test SXP connection switchover only if Controller with SCS is isolated
SxpClusterLib.Check Shards Status
Setup Custom SXP Cluster ${VIRTUAL_IP} listener
- ${controller_index} = SxpClusterLib.Get Active Controller
+ ${controller_index} = SxpClusterLib.Get Leader Controller
Isolate SXP Controller ${controller_index} listener
Isolation of SXP service follower Test Listener Part
[Documentation] Test SXP binding propagation only if Controller with SCS is isolated
SxpClusterLib.Check Shards Status
- ${controller_index} = SxpClusterLib.Get Active Controller
+ ${controller_index} = SxpClusterLib.Get Leader Controller
Setup Custom SXP Cluster ${VIRTUAL_IP} listener
Setup SXP Cluster Bindings ${CLUSTER_NODE_ID} controller${controller_index}
- ${controller_index} = SxpClusterLib.Get Active Controller
+ ${controller_index} = SxpClusterLib.Get Leader Controller
Isolate SXP Controller With Bindings ${controller_index} ${DEVICE_NODE_ID} listener ${DEVICE_SESSION}
Isolation of SXP service follower Test Speaker Part
SxpClusterLib.Check Shards Status
Setup Custom SXP Cluster ${VIRTUAL_IP} speaker
Setup SXP Cluster Bindings ${DEVICE_NODE_ID} ${DEVICE_SESSION}
- ${controller_index} = SxpClusterLib.Get Active Controller
+ ${controller_index} = SxpClusterLib.Get Leader Controller
Isolate SXP Controller With Bindings ${controller_index} ${CLUSTER_NODE_ID} speaker
*** Keywords ***
[Documentation] Prepare topology for testing, creates sessions and generate Route definitions based on Cluster nodes ip
SxpClusterLib.Shutdown Tools Node
SxpClusterLib.Setup SXP Cluster Session
- ${controller_index} = SxpClusterLib.Get Active Controller
+ ${controller_index} = SxpClusterLib.Get Leader Controller
${mac_addresses} = SxpClusterLib.Map Followers To Mac Addresses
BuiltIn.Set Suite Variable ${MAC_ADDRESS_TABLE} ${mac_addresses}
${route} = Sxp.Route Definition Xml ${VIRTUAL_IP} ${VIRTUAL_IP_MASK} ${VIRTUAL_INTERFACE}
Clean Custom SXP Cluster Session
[Documentation] Cleans up resources generated by test
- ${controller_index} = SxpClusterLib.Get Active Controller
+ ${controller_index} = SxpClusterLib.Get Leader Controller
SxpLib.Clean Routing Configuration To Controller controller${controller_index}
SxpClusterLib.Clean SXP Cluster Session
... ip=${EMPTY}
${cluster_mode} = Sxp.Get Opposing Mode ${peer_mode}
SxpLib.Add Connection version4 ${peer_mode} ${peer_address} 64999 ${DEVICE_NODE_ID} session=${DEVICE_SESSION}
- ${controller_id} = SxpClusterLib.Get Active Controller
+ ${controller_id} = SxpClusterLib.Get Leader Controller
SxpLib.Add Node ${CLUSTER_NODE_ID} ip=${peer_address} session=controller${controller_id}
BuiltIn.Wait Until Keyword Succeeds 20 1 SxpClusterLib.Check Cluster Node started ${CLUSTER_NODE_ID}
SxpLib.Add Connection version4 ${cluster_mode} ${TOOLS_SYSTEM_IP} 64999 ${CLUSTER_NODE_ID} session=controller${controller_id}
[Arguments] ${controller_index} ${peer_mode}
[Documentation] Isolate one of cluster nodes and perform check that Device is still connected then revert isolation (and check connection again).
${cluster_mode} = Sxp.Get Opposing Mode ${peer_mode}
- ClusterManagement.Isolate_Member_From_List_Or_All ${controller_index}
+ @{running_members} = ClusterManagement.Isolate_Member_From_List_Or_All ${controller_index}
BuiltIn.Wait Until Keyword Succeeds 240 1 ClusterManagement.Sync_Status_Should_Be_False ${controller_index}
BuiltIn.Wait Until Keyword Succeeds 240 1 SxpClusterLib.Ip Addres Should Not Be Routed To Follower ${MAC_ADDRESS_TABLE} ${VIRTUAL_IP} ${controller_index}
- ${active_follower} = SxpClusterLib.Get Active Controller
+ ${active_follower} = SxpClusterLib.Get Leader Controller From Running @{running_members}
BuiltIn.Wait Until Keyword Succeeds 240 1 SXpClusterLib.Ip Addres Should Be Routed To Follower ${MAC_ADDRESS_TABLE} ${VIRTUAL_IP} ${active_follower}
BuiltIn.Wait Until Keyword Succeeds 60 1 SxpClusterLib.Check Cluster is Connected ${CLUSTER_NODE_ID} mode=${cluster_mode} session=controller${active_follower}
BuiltIn.Wait Until Keyword Succeeds 60 1 Check Device is Connected ${DEVICE_NODE_ID} ${VIRTUAL_IP} ${peer_mode}
${find_session} = BuiltIn.Set Variable If '${session}' == '${EMPTY}' ${True} ${False}
${cluster_mode} = Sxp.Get Opposing Mode ${peer_mode}
${session} = BuiltIn.Set Variable If ${find_session} controller${controller_index} ${session}
- ClusterManagement.Isolate_Member_From_List_Or_All ${controller_index}
+ @{running_members} = ClusterManagement.Isolate_Member_From_List_Or_All ${controller_index}
BuiltIn.Wait Until Keyword Succeeds 240 1 ClusterManagement.Sync_Status_Should_Be_False ${controller_index}
BuiltIn.Wait Until Keyword Succeeds 240 1 SxpClusterLib.Ip Addres Should Not Be Routed To Follower ${MAC_ADDRESS_TABLE} ${VIRTUAL_IP} ${controller_index}
- ${active_follower} = SxpClusterLib.Get Active Controller
+ ${active_follower} = SxpClusterLib.Get Leader Controller From Running @{running_members}
BuiltIn.Wait Until Keyword Succeeds 240 1 SXpClusterLib.Ip Addres Should Be Routed To Follower ${MAC_ADDRESS_TABLE} ${VIRTUAL_IP} ${active_follower}
BuiltIn.Wait Until Keyword Succeeds 60 1 SxpClusterLib.Check Cluster is Connected ${CLUSTER_NODE_ID} mode=${cluster_mode} session=controller${active_follower}
BuiltIn.Wait Until Keyword Succeeds 60 1 Check Device is Connected ${DEVICE_NODE_ID} ${VIRTUAL_IP} ${peer_mode}
Isolation of SXP service follower Test
[Documentation] Test SXP connection switchover only if Controller with SCS is isolated
SxpClusterLib.Check Shards Status
- ${controller_index} = SxpClusterLib.Get Active Controller
+ ${controller_index} = SxpClusterLib.Get Leader Controller
Isolate SXP Controller ${controller_index}
Isolation of SXP noservice follower Test
[Documentation] Test SXP binding propagation only if Controller with SCS is isolated
SxpClusterLib.Check Shards Status
Setup Custom SXP Cluster listener ${CLUSTER_NODE_ID} controller1
- ${controller_index} = SxpClusterLib.Get Active Controller
+ ${controller_index} = SxpClusterLib.Get Leader Controller
Isolate SXP Controller ${controller_index} ${DEVICE_NODE_ID} ${DEVICE_SESSION}
Isolation of SXP service follower Test Speaker Part
[Documentation] Test SXP binding propagation only if Controller with SCS is isolated
SxpClusterLib.Check Shards Status
Setup Custom SXP Cluster speaker
- ${controller_index} = SxpClusterLib.Get Active Controller
+ ${controller_index} = SxpClusterLib.Get Leader Controller
Isolate SXP Controller ${controller_index} ${CLUSTER_NODE_ID}
Isolation of SXP noservice follower Test Listener Part
[Documentation] Isolate one of cluster nodes and perform check that bindings were propagated afterwards reverts isolation
${find_session} = BuiltIn.Set Variable If '${session}' == '' ${True} ${False}
${session} = BuiltIn.Set Variable If ${find_session} controller${controller_index} ${session}
- ClusterManagement.Isolate_Member_From_List_Or_All ${controller_index}
+ @{running_members} = ClusterManagement.Isolate_Member_From_List_Or_All ${controller_index}
BuiltIn.Wait Until Keyword Succeeds 240 1 ClusterManagement.Sync_Status_Should_Be_False ${controller_index}
BuiltIn.Wait Until Keyword Succeeds 60 1 SxpClusterLib.Check Device is Connected ${DEVICE_NODE_ID} session=${DEVICE_SESSION}
- ${active_controller} = SxpClusterLib.Get Active Controller
+ ${active_controller} = SxpClusterLib.Get Leader Controller From Running @{running_members}
${session} = BuiltIn.Set Variable If ${find_session} controller${active_controller} ${session}
BuiltIn.Wait Until Keyword Succeeds 30 1 Check Bindings ${node} ${session}
ClusterManagement.Flush_Iptables_From_List_Or_All
Isolation of RCP service Test
[Documentation] Test SXP RPC functionality only if Controller with SCS is isolated
SxpClusterLib.Check Shards Status
- ${controller_index} = SxpClusterLib.Get Active Controller
+ ${controller_index} = SxpClusterLib.Get Leader Controller
Isolate SXP Controller ${controller_index}
Isolation of RPC noservice Test
Isolate SXP Controller
[Arguments] ${controller_index}
[Documentation] Isolate one of cluster nodes and perform check that RPC changes were performed afterwards reverts isolation
- ${active_controller} = SxpClusterLib.Get Active Controller
+ ${active_controller} = SxpClusterLib.Get Leader Controller
: FOR ${i} IN RANGE ${NUM_ODL_SYSTEM}
\ SxpLib.Add Bindings ${i+1}0 ${i+1}0.${i+1}0.${i+1}0.${i+1}0/32 node=${CLUSTER_NODE_ID} session=controller${active_controller}
- ClusterManagement.Isolate_Member_From_List_Or_All ${controller_index}
+ @{running_members} = ClusterManagement.Isolate_Member_From_List_Or_All ${controller_index}
BuiltIn.Wait Until Keyword Succeeds 240 1 ClusterManagement.Sync_Status_Should_Be_False ${controller_index}
BuiltIn.Wait Until Keyword Succeeds 30 1 Check Bindings Exist
- ${active_controller} = SxpClusterLib.Get Active Controller
+ ${active_controller} = SxpClusterLib.Get Leader Controller From Running @{running_members}
: FOR ${i} IN RANGE ${NUM_ODL_SYSTEM}
\ SXpLib.Delete Bindings ${i+1}0 ${i+1}0.${i+1}0.${i+1}0.${i+1}0/32 node=${CLUSTER_NODE_ID} session=controller${active_controller}
ClusterManagement.Flush_Iptables_From_List_Or_All
Check Bindings Exist
[Documentation] Check that bindings exists in Cluster datastore
- ${controller_index} = SxpClusterLib.Get Active Controller
+ ${controller_index} = SxpClusterLib.Get Leader Controller
${resp} = SxpLib.Get Bindings node=${CLUSTER_NODE_ID} session=controller${controller_index}
: FOR ${i} IN RANGE ${NUM_ODL_SYSTEM}
\ SxpLib.Should Contain Binding ${resp} ${i+1}0 ${i+1}0.${i+1}0.${i+1}0.${i+1}0/32
Check Bindings Does Not Exist
[Documentation] Check that bindings does not exist in Cluster datastore
- ${controller_index} = SxpClusterLib.Get Active Controller
+ ${controller_index} = SxpClusterLib.Get Leader Controller
${resp} = SxpLib.Get Bindings node=${CLUSTER_NODE_ID} session=controller${controller_index}
: FOR ${i} IN RANGE ${NUM_ODL_SYSTEM}
\ SxpLib.Should Not Contain Binding ${resp} ${i+1}0 ${i+1}0.${i+1}0.${i+1}0.${i+1}0/32