*** Settings ***
-Suite Setup Create Controller Sessions
-Suite Teardown Delete All Sessions
-Library RequestsLibrary
-Resource ../../../libraries/ClusterKeywords.robot
-Resource ../../../libraries/MininetKeywords.robot
-Variables ../../../variables/Variables.py
-
-*** Variables ***
-${config_table_0} ${CONFIG_NODES_API}/node/openflow:1/table/0
-${operational_table_0} ${OPERATIONAL_NODES_API}/node/openflow:1/table/0
-${operational_port_1} ${OPERATIONAL_NODES_API}/node/openflow:1/node-connector/openflow:1:1
+Documentation Test suite for Cluster HA - Device Owner failover
+
+Library RequestsLibrary
+Resource ../../../libraries/SetupUtils.robot
+Resource ../../../libraries/ClusterOpenFlow.robot
+Resource ../../../libraries/MininetKeywords.robot
+Resource ../../../libraries/ClusterManagement.robot
+Variables ../../../variables/Variables.py
+
+Suite Setup SetupUtils.Setup_Utils_For_Setup_And_Teardown
+Suite Teardown Delete All Sessions
+
*** Test Cases ***
-Check OpenFlow Shards Status
- [Documentation] Create original cluster list and Check Status for all shards in OpenFlow application.
- ${original_cluster_list} Create Controller Index List
- Set Suite Variable ${original_cluster_list}
- ${inv_conf_leader} ${inv_conf_followers_list} Get Cluster Shard Status ${original_cluster_list} config inventory
- ${inv_oper_leader} ${inv_oper_followers_list} Get Cluster Shard Status ${original_cluster_list} operational inventory
- ${topo_oper_leader} ${topo_oper_followers_list} Get Cluster Shard Status ${original_cluster_list} operational topology
- Log config inventory Leader is ${inv_conf_leader} and followers are ${inv_conf_followers_list}
- Log operational inventory Leader is ${inv_oper_leader} and followers are ${inv_oper_followers_list}
- Log operational topology Leader is ${topo_oper_leader} and followers are ${topo_oper_followers_list}
+Check Shards Status Before Fail
+ [Documentation] Check Status for all shards in OpenFlow application.
+ ClusterOpenFlow.Check OpenFlow Shards Status
Start Mininet Multiple Connections
- [Documentation] Start mininet with connection to all cluster instances.
- ${mininet_conn_id}= Start Mininet Multiple Controllers ${TOOLS_SYSTEM_IP} ${original_cluster_list}
- Set Suite Variable ${mininet_conn_id}
-
-Check Entity Owner Status And Find Owner and Candidate
- [Documentation] Check Entity Owner Status and identify owner and candidate.
- ${original_owner} ${original_candidates_list} Wait Until Keyword Succeeds 5s 1s Get Cluster Entity Owner Status ${original_cluster_list}
- ... openflow openflow:1
- ${original_candidate}= Get From List ${original_candidates_list} 0
- Set Suite Variable ${original_owner}
- Set Suite Variable ${original_candidate}
-
-Check Network Operational Information
- [Documentation] Check device is in operational inventory and topology in all cluster instances.
- ... Inventory should show 1x node_id per device 1x node_id per connector. Topology should show 2x node_id per device + 3x node_id per connector.
- ${dictionary} Create Dictionary openflow:1=4
- Wait Until Keyword Succeeds 5s 1s Check Item Occurrence At URI In Cluster ${original_cluster_list} ${dictionary} ${OPERATIONAL_NODES_API}
- ${dictionary} Create Dictionary openflow:1=11
- Wait Until Keyword Succeeds 5s 1s Check Item Occurrence At URI In Cluster ${original_cluster_list} ${dictionary} ${OPERATIONAL_TOPO_API}
-
-Add Flow In Owner and Verify
+ [Documentation] Start mininet tree,2 with connection to all cluster instances.
+ ${mininet_conn_id}= MininetKeywords.Start Mininet Multiple Controllers
+ ... ${TOOLS_SYSTEM_IP}
+ ... ${ClusterManagement__member_index_list}
+ ... --topo tree,2
+ BuiltIn.Set Suite Variable ${mininet_conn_id}
+ BuiltIn.Wait Until Keyword Succeeds 10s 1s OVSDB.Check OVS OpenFlow Connections ${TOOLS_SYSTEM_IP} 9
+
+Check Entity Owner Status And Find Owner and Successor Before Fail
+ [Documentation] Check Entity Owner Status and identify owner and successor for first switch s1.
+ ${original_owner} ${original_successor_list}= ClusterOpenFlow.Get OpenFlow Entity Owner Status For One Device
+ ... openflow:1
+ ... 1
+ ${original_successor}= Collections.Get From List ${original_successor_list} 0
+ BuiltIn.Set Suite Variable ${original_owner}
+ BuiltIn.Set Suite Variable ${original_successor_list}
+ BuiltIn.Set Suite Variable ${original_successor}
+
+Reconnect Extra Switches To Successors And Check OVS Connections
+ [Documentation] Connect switches s2 and s3 to successor instances.
+ ${controller_opt}= BuiltIn.Set Variable
+ FOR ${index} IN @{original_successor_list}
+ ${controller_opt}= BuiltIn.Catenate
+ ... ${controller_opt}
+ ... ${SPACE}tcp:${ODL_SYSTEM_${index}_IP}:${ODL_OF_PORT}
+ Log ${controller_opt}
+ END
+ OVSDB.Set Controller In OVS Bridge ${TOOLS_SYSTEM_IP} s2 ${controller_opt}
+ OVSDB.Set Controller In OVS Bridge ${TOOLS_SYSTEM_IP} s3 ${controller_opt}
+ BuiltIn.Wait Until Keyword Succeeds 10s 1s OVSDB.Check OVS OpenFlow Connections ${TOOLS_SYSTEM_IP} 7
+
+Check Network Operational Information Before Fail
+ [Documentation] Check devices in operational inventory and topology in all cluster instances.
+ ClusterOpenFlow.Check OpenFlow Network Operational Information For Sample Topology
+
+Add Configuration In Owner and Verify Before Fail
+ [Documentation] Add Flow in Owner and verify it gets applied from all instances.
+ ClusterOpenFlow.Add Sample Flow and Verify ${original_owner}
+
+Modify Configuration In Owner and Verify Before Fail
+ [Documentation] Modify Flow in Owner and verify it gets applied from all instances.
+ ClusterOpenFlow.Modify Sample Flow and Verify ${original_owner}
+
+Delete Configuration In Owner and Verify Before Fail
+ [Documentation] Delete Flow in Owner and verify it gets applied from all instances.
+ ClusterOpenFlow.Delete Sample Flow and Verify ${original_owner}
+
+Add Configuration In Successor and Verify Before Fail
+ [Documentation] Add Flow in Successor and verify it gets applied from all instances.
+ ClusterOpenFlow.Add Sample Flow and Verify ${original_successor}
+
+Modify Configuration In Successor and Verify Before Fail
+ [Documentation] Modify Flow in successor and verify it gets applied from all instances.
+ ClusterOpenFlow.Modify Sample Flow and Verify ${original_successor}
+
+Delete Configuration In Successor and Verify Before Fail
+ [Documentation] Delete Flow in Successor and verify it gets removed from all instances.
+ ClusterOpenFlow.Delete Sample Flow and Verify ${original_successor}
+
+Send RPC Add to Owner and Verify Before Fail
+ [Documentation] Add Flow in Owner and verify it gets applied from all instances.
+ ClusterOpenFlow.Send RPC Add Sample Flow and Verify ${original_owner}
+
+Send RPC Delete to Owner and Verify Before Fail
+ [Documentation] Delete Flow in Owner and verify it gets removed from all instances.
+ ClusterOpenFlow.Send RPC Delete Sample Flow and Verify ${original_owner}
+
+Send RPC Add to Successor and Verify Before Fail
+ [Documentation] Add Flow in Successor and verify it gets applied from all instances.
+ ClusterOpenFlow.Send RPC Add Sample Flow and Verify ${original_successor}
+
+Send RPC Delete to Successor and Verify Before Fail
+ [Documentation] Delete Flow in Successor and verify it gets removed from all instances.
+ ClusterOpenFlow.Send RPC Delete Sample Flow and Verify ${original_successor}
+
+Modify Network And Verify Before Fail
+ [Documentation] Take a link down and verify port status in all instances.
+ ClusterOpenFlow.Take OpenFlow Device Link Down and Verify
+
+Restore Network And Verify Before Fail
+ [Documentation] Take the link up and verify port status in all instances.
+ ClusterOpenFlow.Take OpenFlow Device Link Up and Verify
+
+Kill Owner Instance
+ [Documentation] Kill Owner Instance and verify it is dead
+ ClusterManagement.Kill Single Member ${original_owner}
+ BuiltIn.Set Suite Variable ${new_cluster_list} ${original_successor_list}
+
+Check Shards Status After Fail
+ [Documentation] Create original cluster list and check Status for all shards in OpenFlow application.
+ ClusterOpenFlow.Check OpenFlow Shards Status After Cluster Event ${new_cluster_list}
+
+Check Entity Owner Status And Find Owner and Successor After Fail
+ [Documentation] Check Entity Owner Status and identify owner and successor.
+ ${new_owner} ${new_successor_list}= ClusterOpenFlow.Get OpenFlow Entity Owner Status For One Device
+ ... openflow:1
+ ... ${original_successor}
+ ... ${new_cluster_list}
+ ${new_successor}= Collections.Get From List ${new_successor_list} 0
+ BuiltIn.Set Suite Variable ${new_owner}
+ BuiltIn.Set Suite Variable ${new_successor}
+
+Check Network Operational Information After Fail
+ [Documentation] Check devices in operational inventory and topology in all cluster instances.
+ ClusterOpenFlow.Check OpenFlow Network Operational Information For Sample Topology ${new_cluster_list}
+
+Add Configuration In Owner and Verify After Fail
+ [Documentation] Add Flow in Owner and verify it gets applied from all instances.
+ ClusterOpenFlow.Add Sample Flow and Verify ${new_owner} ${new_cluster_list}
+
+Modify Configuration In Owner and Verify After Fail
+ [Documentation] Modify Flow in Owner and verify it gets applied from all instances.
+ ClusterOpenFlow.Modify Sample Flow and Verify ${new_owner} ${new_cluster_list}
+
+Delete Configuration In Owner and Verify After Fail
+ [Documentation] Delete Flow in Owner and verify it gets applied from all instances.
+ ClusterOpenFlow.Delete Sample Flow and Verify ${new_owner} ${new_cluster_list}
+
+Add Configuration In Successor and Verify After Fail
+ [Documentation] Add Flow in Successor and verify it gets applied from all instances.
+ ClusterOpenFlow.Add Sample Flow and Verify ${new_successor} ${new_cluster_list}
+
+Modify Configuration In Successor and Verify After Fail
+ [Documentation] Modify Flow in Successor and verify it gets applied from all instances.
+ ClusterOpenFlow.Modify Sample Flow and Verify ${new_successor} ${new_cluster_list}
+
+Delete Configuration In Successor and Verify After Fail
+ [Documentation] Delete Flow in Successor and verify it gets applied from all instances.
+ ClusterOpenFlow.Delete Sample Flow and Verify ${new_successor} ${new_cluster_list}
+
+Send RPC Add to Owner and Verify After Fail
[Documentation] Add Flow in Owner and verify it gets applied from all instances.
- ${body}= OperatingSystem.Get File ${CURDIR}/../../../variables/openflowplugin/sample_flow_1.json
- ${dictionary}= Create Dictionary 10.0.1.0/24=1
- Put And Check At URI In Cluster ${original_cluster_list} ${original_owner} ${config_table_0}/flow/1 ${body} ${HEADERS}
- Wait Until Keyword Succeeds 10s 1s Check Item Occurrence At URI In Cluster ${original_cluster_list} ${dictionary} ${operational_table_0}
+ ClusterOpenFlow.Send RPC Add Sample Flow and Verify ${new_owner} ${new_cluster_list}
+
+Send RPC Delete to Owner and Verify After Fail
+ [Documentation] Delete Flow in Owner and verify it gets removed from all instances.
+ ClusterOpenFlow.Send RPC Delete Sample Flow and Verify ${new_owner} ${new_cluster_list}
+
+Send RPC Add to Successor and Verify After Fail
+ [Documentation] Add Flow in Successor and verify it gets applied from all instances.
+ ClusterOpenFlow.Send RPC Add Sample Flow and Verify ${new_successor} ${new_cluster_list}
+
+Send RPC Delete to Successor and Verify After Fail
+ [Documentation] Delete Flow in Successor and verify it gets removed from all instances.
+ ClusterOpenFlow.Send RPC Delete Sample Flow and Verify ${new_successor} ${new_cluster_list}
+
+Modify Network and Verify After Fail
+ [Documentation] Take a link down and verify port status in all instances.
+ ClusterOpenFlow.Take OpenFlow Device Link Down and Verify ${new_cluster_list}
+
+Restore Network and Verify After Fail
+ [Documentation] Take the link up and verify port status in all instances.
+ ClusterOpenFlow.Take OpenFlow Device Link Up and Verify ${new_cluster_list}
-Modify Flow In Owner and Verify
+Start Old Owner Instance
+ [Documentation] Start old Owner Instance and verify it is up
+ ClusterManagement.Start Single Member ${original_owner}
+ [Teardown] Utils.Report Failure Due To Bug CONTROLLER-1849
+
+Check Shards Status After Recover
+ [Documentation] Create original cluster list and check Status for all shards in OpenFlow application.
+ ClusterOpenFlow.Check OpenFlow Shards Status After Cluster Event
+
+Check Entity Owner Status After Recover
+ [Documentation] Check Entity Owner Status and identify owner and successor.
+ ${new_owner} ${new_successors_list}= ClusterOpenFlow.Get OpenFlow Entity Owner Status For One Device
+ ... openflow:1
+ ... 1
+ BuiltIn.Set Suite Variable ${new_owner}
+
+Check Network Operational Information After Recover
+ [Documentation] Check devices in operational inventory and topology in all cluster instances.
+ ClusterOpenFlow.Check OpenFlow Network Operational Information For Sample Topology
+
+Add Configuration In Owner and Verify After Recover
+ [Documentation] Add Flow in Owner and verify it gets applied from all instances.
+ ClusterOpenFlow.Add Sample Flow and Verify ${new_owner}
+
+Modify Configuration In Owner and Verify After Recover
[Documentation] Modify Flow in Owner and verify it gets applied from all instances.
- ${body}= OperatingSystem.Get File ${CURDIR}/../../../variables/openflowplugin/sample_flow_2.json
- ${dictionary}= Create Dictionary 10.0.2.0/24=1
- Put And Check At URI In Cluster ${original_cluster_list} ${original_owner} ${config_table_0}/flow/1 ${body} ${HEADERS}
- Wait Until Keyword Succeeds 10s 1s Check Item Occurrence At URI In Cluster ${original_cluster_list} ${dictionary} ${operational_table_0}
+ ClusterOpenFlow.Modify Sample Flow and Verify ${new_owner}
-Delete Flow In Owner and Verify
+Delete Configuration In Owner and Verify After Recover
[Documentation] Delete Flow in Owner and verify it gets applied from all instances.
- ${dictionary}= Create Dictionary 10.0.2.0/24=0
- Delete And Check At URI In Cluster ${original_cluster_list} ${original_owner} ${config_table_0}/flow/1
- Wait Until Keyword Succeeds 10s 1s Check Item Occurrence At URI In Cluster ${original_cluster_list} ${dictionary} ${operational_table_0}
+ ClusterOpenFlow.Delete Sample Flow and Verify ${new_owner}
-Add Flow In Candidate and Verify
+Add Configuration In Old Owner and Verify After Recover
[Documentation] Add Flow in Owner and verify it gets applied from all instances.
- ${body}= OperatingSystem.Get File ${CURDIR}/../../../variables/openflowplugin/sample_flow_1.json
- ${dictionary}= Create Dictionary 10.0.1.0/24=1
- Put And Check At URI In Cluster ${original_cluster_list} ${original_candidate} ${config_table_0}/flow/1 ${body} ${HEADERS}
- Wait Until Keyword Succeeds 10s 1s Check Item Occurrence At URI In Cluster ${original_cluster_list} ${dictionary} ${operational_table_0}
+ ClusterOpenFlow.Add Sample Flow and Verify ${original_owner}
-Modify Flow In Candidate and Verify
+Modify Configuration In Old Owner and Verify After Recover
[Documentation] Modify Flow in Owner and verify it gets applied from all instances.
- ${body}= OperatingSystem.Get File ${CURDIR}/../../../variables/openflowplugin/sample_flow_2.json
- ${dictionary}= Create Dictionary 10.0.2.0/24=1
- Put And Check At URI In Cluster ${original_cluster_list} ${original_candidate} ${config_table_0}/flow/1 ${body} ${HEADERS}
- Wait Until Keyword Succeeds 10s 1s Check Item Occurrence At URI In Cluster ${original_cluster_list} ${dictionary} ${operational_table_0}
+ ClusterOpenFlow.Modify Sample Flow and Verify ${original_owner}
-Delete Flow In Candidate and Verify
+Delete Configuration In Old Owner and Verify After Recover
+ [Documentation] Delete Flow in Owner and verify it gets applied from all instances.
+ ClusterOpenFlow.Delete Sample Flow and Verify ${original_owner}
+
+Send RPC Add to Owner and Verify After Recover
+ [Documentation] Add Flow in Owner and verify it gets applied from all instances.
+ ClusterOpenFlow.Send RPC Add Sample Flow and Verify ${new_owner}
+
+Send RPC Delete to Owner and Verify After Recover
[Documentation] Delete Flow in Owner and verify it gets removed from all instances.
- ${dictionary}= Create Dictionary 10.0.2.0/24=0
- Delete And Check At URI In Cluster ${original_cluster_list} ${original_candidate} ${config_table_0}/flow/1
- Wait Until Keyword Succeeds 10s 1s Check Item Occurrence At URI In Cluster ${original_cluster_list} ${dictionary} ${operational_table_0}
+ ClusterOpenFlow.Send RPC Delete Sample Flow and Verify ${new_owner}
-Send RPC Add Flow to Owner and Verify
+Send RPC Add to Old Owner and Verify After Recover
[Documentation] Add Flow in Owner and verify it gets applied from all instances.
- ${body}= OperatingSystem.Get File ${CURDIR}/../../../variables/openflowplugin/add_flow_rpc.json
- ${dictionary}= Create Dictionary 10.0.1.0/24=1
- ${resp} RequestsLibrary.Post Request controller${original_owner} /restconf/operations/sal-flow:add-flow ${body} ${HEADERS}
- Log ${resp.content}
- Should Be Equal As Strings ${resp.status_code} 200
- Wait Until Keyword Succeeds 10s 1s Check Item Occurrence At URI In Cluster ${original_cluster_list} ${dictionary} ${operational_table_0}
-
-Send RPC Delete Flow to Owner and Verify
+ ClusterOpenFlow.Send RPC Add Sample Flow and Verify ${original_owner}
+
+Send RPC Delete to Old Owner and Verify After Recover
[Documentation] Delete Flow in Owner and verify it gets removed from all instances.
- ${body}= OperatingSystem.Get File ${CURDIR}/../../../variables/openflowplugin/delete_flow_rpc.json
- ${dictionary}= Create Dictionary 10.0.1.0/24=0
- ${resp} RequestsLibrary.Post Request controller${original_owner} /restconf/operations/sal-flow:remove-flow ${body} ${HEADERS}
- Log ${resp.content}
- Should Be Equal As Strings ${resp.status_code} 200
- Wait Until Keyword Succeeds 10s 1s Check Item Occurrence At URI In Cluster ${original_cluster_list} ${dictionary} ${operational_table_0}
-
-Send RPC Add Flow to Candidate and Verify
- [Documentation] Add Flow in Candidate and verify it gets applied from all instances.
- ${body}= OperatingSystem.Get File ${CURDIR}/../../../variables/openflowplugin/add_flow_rpc.json
- ${dictionary}= Create Dictionary 10.0.1.0/24=1
- ${resp} RequestsLibrary.Post Request controller${original_candidate} /restconf/operations/sal-flow:add-flow ${body} ${HEADERS}
- Log ${resp.content}
- Should Be Equal As Strings ${resp.status_code} 200
- Wait Until Keyword Succeeds 10s 1s Check Item Occurrence At URI In Cluster ${original_cluster_list} ${dictionary} ${operational_table_0}
-
-Send RPC Delete Flow to Candidate and Verify
- [Documentation] Delete Flow in Candidate and verify it gets removed from all instances.
- ${body}= OperatingSystem.Get File ${CURDIR}/../../../variables/openflowplugin/delete_flow_rpc.json
- ${dictionary}= Create Dictionary 10.0.1.0/24=0
- ${resp} RequestsLibrary.Post Request controller${original_candidate} /restconf/operations/sal-flow:remove-flow ${body} ${HEADERS}
- Log ${resp.content}
- Should Be Equal As Strings ${resp.status_code} 200
- Wait Until Keyword Succeeds 10s 1s Check Item Occurrence At URI In Cluster ${original_cluster_list} ${dictionary} ${operational_table_0}
-
-Take a Link Down and Verify
+ ClusterOpenFlow.Send RPC Delete Sample Flow and Verify ${original_owner}
+
+Modify Network and Verify After Recover
[Documentation] Take a link down and verify port status in all instances.
- ${dictionary}= Create Dictionary "link-down":true=1
- ${ouput}= Send Mininet Command ${mininet_conn_id} link s1 h1 down
- Wait Until Keyword Succeeds 5s 1s Check Item Occurrence At URI In Cluster ${original_cluster_list} ${dictionary} ${operational_port_1}
+ ClusterOpenFlow.Take OpenFlow Device Link Down and Verify
-Take a Link Up and Verify
+Restore Network and Verify After Recover
[Documentation] Take the link up and verify port status in all instances.
- ${dictionary}= Create Dictionary "link-down":true=0
- ${ouput}= Send Mininet Command ${mininet_conn_id} link s1 h1 up
- Wait Until Keyword Succeeds 5s 1s Check Item Occurrence At URI In Cluster ${original_cluster_list} ${dictionary} ${operational_port_1}
+ ClusterOpenFlow.Take OpenFlow Device Link Up and Verify
Stop Mininet and Exit
[Documentation] Stop mininet and exit connection.
- Stop Mininet And Exit ${mininet_conn_id}
- Clean Mininet System
+ MininetKeywords.Stop Mininet And Exit ${mininet_conn_id}
+ Utils.Clean Mininet System
Check No Network Operational Information
[Documentation] Check device is not in operational inventory or topology in all cluster instances.
- ${dictionary} Create Dictionary openflow:1=0
- Wait Until Keyword Succeeds 5s 1s Check Item Occurrence At URI In Cluster ${original_cluster_list} ${dictionary} ${OPERATIONAL_NODES_API}
- ${dictionary} Create Dictionary openflow:1=0
- Wait Until Keyword Succeeds 5s 1s Check Item Occurrence At URI In Cluster ${original_cluster_list} ${dictionary} ${OPERATIONAL_TOPO_API}
+ ClusterOpenFlow.Check No OpenFlow Network Operational Information
Code Review / integration / test.git / blobdiff