*** Settings *** Documentation Switch connections and cluster are restarted. Library RequestsLibrary Library Collections Resource ../../../libraries/SetupUtils.robot Resource ../../../libraries/ClusterManagement.robot Resource ../../../libraries/ClusterOpenFlow.robot Resource ../../../libraries/TemplatedRequests.robot Resource ../../../libraries/MininetKeywords.robot Resource ../../../libraries/Utils.robot Resource ../../../libraries/OvsManager.robot Resource ../../../variables/Variables.robot Resource ../../../variables/openflowplugin/Variables.robot Suite Setup Initialization Phase Suite Teardown Final Phase Test Setup SetupUtils.Setup_Test_With_Logging_And_Without_Fast_Failing *** Variables *** ${SWITCHES} 3 ${ITER} 100 ${VAR_DIR} ${CURDIR}/../../../variables/openflowplugin *** Test Cases *** Add Groups And Flows [Documentation] Add ${ITER} groups 1&2 and flows in every switch. Add Groups And Flows On Member ${ITER} Start Mininet Multiple Connections [Documentation] Start mininet linear with connection to all cluster instances. ${cluster_index_list}= ClusterManagement.List All Indices ${mininet_conn_id}= MininetKeywords.Start Mininet Multiple Controllers ... ${TOOLS_SYSTEM_IP} ... ${cluster_index_list} ... --topo linear,${SWITCHES} BuiltIn.Set Suite Variable ${cluster_index_list} BuiltIn.Set Suite Variable ${mininet_conn_id} BuiltIn.Wait Until Keyword Succeeds ... 10s ... 1s ... OVSDB.Check OVS OpenFlow Connections ... ${TOOLS_SYSTEM_IP} ... ${SWITCHES*3} Check Linear Topology [Documentation] Check Linear Topology. BuiltIn.Wait Until Keyword Succeeds 30s 1s ClusterOpenFlow.Check Linear Topology On Member ${SWITCHES} Check Stats Are Not Frozen [Documentation] Check that duration flow stat is increasing. BuiltIn.Wait Until Keyword Succeeds 30s 1s Check Flow Stats Are Not Frozen Check Flows In Operational DS [Documentation] Check Flows in operational DS. BuiltIn.Wait Until Keyword Succeeds 10s 1s ClusterOpenFlow.Check Number Of Flows On Member ${all_flows} Check Groups In Operational DS [Documentation] Check Groups in operational DS. BuiltIn.Wait Until Keyword Succeeds ... 10s ... 1s ... ClusterOpenFlow.Check Number Of Groups On Member ... ${all_groups} Check Flows In Switch [Documentation] Check Flows in switch. MininetKeywords.Check Flows In Mininet ${mininet_conn_id} ${all_flows} Check Entity Owner Status And Find Owner and Successor Before Fail [Documentation] Check Entity Owner Status and identify owner and successor for first switch s1. ${original_owner} ${original_successor_list}= ClusterOpenFlow.Get OpenFlow Entity Owner Status For One Device ... openflow:1 ... 1 BuiltIn.Set Suite Variable ${original_owner} BuiltIn.Set Suite Variable ${new_cluster_list} ${original_successor_list} Disconnect Mininet From Owner [Documentation] Disconnect mininet from the owner ${original_owner_list}= BuiltIn.Create List ${original_owner} MininetKeywords.Disconnect Cluster Mininet break ${original_owner_list} BuiltIn.Set Suite Variable ${original_owner_list} Check Entity Owner Status And Find Owner and Successor After Fail [Documentation] Check Entity Owner Status and identify owner and successor for first switch s1. ${new_owner} ${new_successor_list}= BuiltIn.Wait Until Keyword Succeeds ... 10s ... 1s ... ClusterOpenFlow.Get OpenFlow Entity Owner Status For One Device ... openflow:1 ... 1 ... ${new_cluster_list} ... after_stop=True ${owner_list}= BuiltIn.Create List ${original_owner} ${new_owner} BuiltIn.Set Suite Variable ${owner_list} BuiltIn.Set Suite Variable ${new_owner} BuiltIn.Set Suite Variable ${new_successor_list} Check Switch Moves To New Master [Documentation] Check switch s1 is connected to new Master. ${new_master}= BuiltIn.Set Variable ${ODL_SYSTEM_${new_owner}_IP} BuiltIn.Wait Until Keyword Succeeds ... 10s ... 1s ... OvsManager.Should Be Master ... s1 ... ${new_master} ... update_data=${True} Check Linear Topology After Disconnect [Documentation] Check Linear Topology. BuiltIn.Wait Until Keyword Succeeds 30s 1s ClusterOpenFlow.Check Linear Topology On Member ${SWITCHES} Check Stats Are Not Frozen After Disconnect [Documentation] Check that duration flow stat is increasing BuiltIn.Wait Until Keyword Succeeds 30s 1s Check Flow Stats Are Not Frozen Remove Flows And Groups After Mininet Is Disconnected [Documentation] Remove 1 group 1&2 and 1 flow in every switch. Remove Single Group And Flow On Member Check Flows In Operational DS After Mininet Is Disconnected [Documentation] Check Flows in Operational DS. BuiltIn.Wait Until Keyword Succeeds ... 30s ... 1s ... ClusterOpenFlow.Check Number Of Flows On Member ... ${less_flows} Check Groups In Operational DS After Mininet Is Disconnected [Documentation] Check Groups in Operational DS. BuiltIn.Wait Until Keyword Succeeds ... 10s ... 1s ... ClusterOpenFlow.Check Number Of Groups On Member ... ${less_groups} Check Flows In Switch After Mininet Is Disconnected [Documentation] Check Flows in switch. MininetKeywords.Check Flows In Mininet ${mininet_conn_id} ${less_flows} Reconnect Mininet To Owner [Documentation] Reconnect mininet to switch 1 owner. MininetKeywords.Disconnect Cluster Mininet restore ${original_owner_list} Check Entity Owner Status And Find Owner and Successor After Reconnect [Documentation] Check Entity Owner Status and identify owner and successor for first switch s1. BuiltIn.Wait Until Keyword Succeeds ... 10s ... 1s ... ClusterOpenFlow.Get OpenFlow Entity Owner Status For One Device ... openflow:1 ... 1 Add Flows And Groups After Owner Reconnect [Documentation] Add 1 group type 1&2 and 1 flow in every switch. Add Single Group And Flow On Member Check Stats Are Not Frozen After Owner Reconnect [Documentation] Check that duration flow stat is increasing. BuiltIn.Wait Until Keyword Succeeds 30s 1s Check Flow Stats Are Not Frozen Check Flows After Owner Reconnect In Operational DS [Documentation] Check Flows in Operational DS. BuiltIn.Wait Until Keyword Succeeds 30s 1s ClusterOpenFlow.Check Number Of Flows On Member ${all_flows} Check Groups After Owner Reconnect In Operational DS [Documentation] Check Groups in Operational DS. BuiltIn.Wait Until Keyword Succeeds ... 10s ... 1s ... ClusterOpenFlow.Check Number Of Groups On Member ... ${all_groups} Check Flows After Owner Reconnect In Switch [Documentation] Check Flows in switch. MininetKeywords.Check Flows In Mininet ${mininet_conn_id} ${all_flows} Check Switches Generate Slave Connection [Documentation] Check switches are connected to new Slave. ${original_master}= BuiltIn.Set Variable ${ODL_SYSTEM_${original_owner}_IP} FOR ${switch} IN RANGE 1 ${switches+1} BuiltIn.Wait Until Keyword Succeeds ... 50s ... 1s ... OvsManager.Should Be Slave ... s${switch} ... ${original_master} ... update_data=${True} END Disconnect Mininet From Successor [Documentation] Disconnect mininet from the Successor. MininetKeywords.Disconnect Cluster Mininet break ${new_successor_list} Check Entity Owner Status And Find New Owner and Successor After Disconnect [Documentation] Check Entity Owner Status and identify owner and successor for first switch s1. ${current_owner} ${current_successor_list}= BuiltIn.Wait Until Keyword Succeeds ... 10s ... 1s ... ClusterOpenFlow.Get OpenFlow Entity Owner Status For One Device ... openflow:1 ... 1 ... ${owner_list} ... after_stop=True BuiltIn.Set Suite Variable ${current_owner} BuiltIn.Set Suite Variable ${current_successor_list} Disconnect Mininet From Current Owner [Documentation] Disconnect mininet from the owner ${current_owner_list}= BuiltIn.Create List ${current_owner} MininetKeywords.Disconnect Cluster Mininet break ${current_owner_list} Check Entity Owner Status And Find Current Owner and Successor After Disconnect [Documentation] Check Entity Owner Status and identify owner and successor for first switch s1. ${current_new_owner} ${current_new_successor_list}= BuiltIn.Wait Until Keyword Succeeds ... 10s ... 1s ... ClusterOpenFlow.Get OpenFlow Entity Owner Status For One Device ... openflow:1 ... 1 ... ${original_owner_list} ... after_stop=True BuiltIn.Set Suite Variable ${current_new_owner} BuiltIn.Set Suite Variable ${current_new_successor_list} Check Switch Moves To Current Master [Documentation] Check switch s1 is connected to original Master. ${current_new_master}= BuiltIn.Set Variable ${ODL_SYSTEM_${current_new_owner}_IP} BuiltIn.Wait Until Keyword Succeeds ... 10s ... 1s ... OvsManager.Should Be Master ... s1 ... ${current_new_master} ... update_data=${True} BuiltIn.Should Be Equal ${current_new_owner} ${original_owner} Check Linear Topology After Owner Disconnect [Documentation] Check Linear Topology. BuiltIn.Wait Until Keyword Succeeds 30s 1s ClusterOpenFlow.Check Linear Topology On Member ${SWITCHES} Check Stats Are Not Frozen After Owner Disconnect [Documentation] Check that duration flow stat is increasing. BuiltIn.Wait Until Keyword Succeeds 30s 1s Check Flow Stats Are Not Frozen Remove Flows And Groups After Owner Disconnected [Documentation] Remove 1 group 1&2 and 1 flow in every switch. Remove Single Group And Flow On Member Check Flows In Operational DS After Owner Disconnected [Documentation] Check Flows in Operational DS. BuiltIn.Wait Until Keyword Succeeds ... 30s ... 1s ... ClusterOpenFlow.Check Number Of Flows On Member ... ${less_flows} Check Groups In Operational DS After Owner Disconnected [Documentation] Check Groups in Operational DS. BuiltIn.Wait Until Keyword Succeeds ... 10s ... 1s ... ClusterOpenFlow.Check Number Of Groups On Member ... ${less_groups} Check Flows In Switch After Owner Disconnected [Documentation] Check Flows in switch. MininetKeywords.Check Flows In Mininet ${mininet_conn_id} ${less_flows} Disconnect Mininet From Cluster [Documentation] Disconnect Mininet from Cluster. MininetKeywords.Disconnect Cluster Mininet break ${original_owner_list} Check No Switches After Disconnect [Documentation] Check no switches in topology. BuiltIn.Wait Until Keyword Succeeds 30s 1s ClusterOpenFlow.Check No Switches On Member ${SWITCHES} Check Switch Is Not Connected [Documentation] Check switch s1 is not connected to any controller. FOR ${index} IN @{cluster_index_list} BuiltIn.Wait Until Keyword Succeeds ... 10s ... 1s ... OvsManager.Should Be Disconnected ... s1 ... ${ODL_SYSTEM_${index}_IP} ... update_data=${True} END Reconnect Mininet To Cluster [Documentation] Reconnect mininet to cluster by removing Iptables drop rules that were used to disconnect MininetKeywords.Disconnect Cluster Mininet restore Check Linear Topology After Mininet Reconnects [Documentation] Check Linear Topology. BuiltIn.Wait Until Keyword Succeeds 10s 1s ClusterOpenFlow.Check Linear Topology On Member ${SWITCHES} Add Flows And Groups After Mininet Reconnects [Documentation] Add 1 group type 1&2 and 1 flow in every switch. Add Single Group And Flow On Member Check Flows In Operational DS After Mininet Reconnects [Documentation] Check Flows in Operational DS. BuiltIn.Wait Until Keyword Succeeds 30s 1s ClusterOpenFlow.Check Number Of Flows On Member ${all_flows} Check Groups In Operational DS After Mininet Reconnects [Documentation] Check Groups in Operational DS. BuiltIn.Wait Until Keyword Succeeds ... 10s ... 1s ... ClusterOpenFlow.Check Number Of Groups On Member ... ${all_groups} Check Flows In Switch After Mininet Reconnects [Documentation] Check Flows in switch. MininetKeywords.Check Flows In Mininet ${mininet_conn_id} ${all_flows} Check Entity Owner Status And Find Owner and Successor Before Owner Stop [Documentation] Check Entity Owner Status and identify owner and successor for first switch s1. ${original_owner} ${original_successor_list}= ClusterOpenFlow.Get OpenFlow Entity Owner Status For One Device ... openflow:1 ... 1 ${original_successor}= Collections.Get From List ${original_successor_list} 0 BuiltIn.Set Suite Variable ${original_owner} BuiltIn.Set Suite Variable ${original_successor_list} BuiltIn.Set Suite Variable ${original_successor} Check Switch Generates Slave Connection Before Owner Stop [Documentation] Check switch s1 is connected to Slave. ${original_slave}= BuiltIn.Set Variable ${ODL_SYSTEM_${original_successor}_IP} BuiltIn.Wait Until Keyword Succeeds ... 50s ... 1s ... OvsManager.Should Be Slave ... s1 ... ${original_slave} ... update_data=${True} Check Shards Status Before Owner Stop [Documentation] Check Status for all shards in OpenFlow application. ClusterOpenFlow.Check OpenFlow Shards Status Stop Owner Instance [Documentation] Stop Owner Instance and verify it is shutdown ClusterManagement.Stop Single Member ${original_owner} BuiltIn.Set Suite Variable ${new_cluster_list} ${original_successor_list} Check Shards Status After Stop [Documentation] Check Status for all shards in OpenFlow application. ClusterOpenFlow.Check OpenFlow Shards Status After Cluster Event ${new_cluster_list} Check Entity Owner Status And Find Owner and Successor After Stop [Documentation] Check Entity Owner Status and identify owner and successor. ${new_owner} ${new_successor_list}= ClusterOpenFlow.Get OpenFlow Entity Owner Status For One Device ... openflow:1 ... ${original_successor} ... ${new_cluster_list} ... after_stop=True BuiltIn.Set Suite Variable ${new_owner} Check Stats Are Not Frozen After Owner Stop [Documentation] Check that duration flow stat is increasing in new owner. BuiltIn.Wait Until Keyword Succeeds 30s 1s Check Flow Stats Are Not Frozen ${new_owner} Remove Configuration In Owner and Verify After Owner Stop [Documentation] Remove 1 group 1&2 and 1 flow in every switch. Remove Single Group And Flow On Member ${new_owner} Check Flows After Owner Stop In Operational DS [Documentation] Check Flows in Operational DS in new owner. BuiltIn.Wait Until Keyword Succeeds ... 30s ... 1s ... ClusterOpenFlow.Check Number Of Flows On Member ... ${less_flows} ... ${new_owner} Check Groups After Owner Stop In Operational DS [Documentation] Check Groups in Operational DS in new owner. BuiltIn.Wait Until Keyword Succeeds ... 10s ... 1s ... ClusterOpenFlow.Check Number Of Groups On Member ... ${less_groups} ... ${new_owner} Check Flows In Switch After Owner Stop [Documentation] Check Flows in switch. MininetKeywords.Check Flows In Mininet ${mininet_conn_id} ${less_flows} Start Old Owner Instance [Documentation] Start old Owner Instance and verify it is up. ClusterManagement.Start Single Member ${original_owner} Check Entity Owner Status And Find Owner and Successor After Start Owner [Documentation] Check Entity Owner Status and identify owner and successor for first switch s1. ${owner} ${successor_list}= BuiltIn.Wait Until Keyword Succeeds ... 10s ... 1s ... ClusterOpenFlow.Get OpenFlow Entity Owner Status For One Device ... openflow:1 ... 1 Check Linear Topology After Owner Restart [Documentation] Check Linear Topology. BuiltIn.Wait Until Keyword Succeeds 10s 1s ClusterOpenFlow.Check Linear Topology On Member ${SWITCHES} Add Configuration In Owner and Verify After Owner Restart [Documentation] Add 1 group type 1&2 and 1 flow in every switch. Add Single Group And Flow On Member ${new_owner} Check Stats Are Not Frozen After Owner Restart [Documentation] Check that duration flow stat is increasing. Log ${new_owner} BuiltIn.Wait Until Keyword Succeeds 30s 1s Check Flow Stats Are Not Frozen Check Flows In Operational DS After Owner Restart [Documentation] Check Flows in Operational DS. BuiltIn.Wait Until Keyword Succeeds 30s 1s ClusterOpenFlow.Check Number Of Flows On Member ${all_flows} Check Groups In Operational DS After Owner Restart [Documentation] Check Groups in Operational DS. BuiltIn.Wait Until Keyword Succeeds ... 10s ... 1s ... ClusterOpenFlow.Check Number Of Groups On Member ... ${all_groups} Check Flows In Switch After Owner Restart [Documentation] Check Flows in switch. MininetKeywords.Check Flows In Mininet ${mininet_conn_id} ${all_flows} Restart Cluster [Documentation] Stop and Start cluster. # Try to stop contoller, if stop does not work or takes too long, kill controller. ${status} ${result}= BuiltIn.Run Keyword And Ignore Error ClusterManagement.Stop_Members_From_List_Or_All IF '${status}' != 'PASS' ClusterManagement.Kill_Members_From_List_Or_All END ClusterManagement.Start_Members_From_List_Or_All wait_for_sync=False Check Linear Topology After Controller Restarts [Documentation] Check Linear Topology BuiltIn.Wait Until Keyword Succeeds 300s 2s ClusterOpenFlow.Check Linear Topology On Member ${SWITCHES} Check Stats Are Not Frozen After Cluster Restart [Documentation] Check that duration flow stat is increasing. BuiltIn.Wait Until Keyword Succeeds 30s 1s Check Flow Stats Are Not Frozen Check Flows In Operational DS After Controller Restarts [Documentation] Check Flows in Operational DS. BuiltIn.Wait Until Keyword Succeeds 30s 1s ClusterOpenFlow.Check Number Of Flows On Member ${all_flows} Check Groups In Operational DS After Controller Restarts [Documentation] Check Groups in Operational DS. BuiltIn.Wait Until Keyword Succeeds ... 10s ... 1s ... ClusterOpenFlow.Check Number Of Groups On Member ... ${all_groups} Check Flows In Switch After Controller Restarts [Documentation] Check Flows in switch. MininetKeywords.Check Flows In Mininet ${mininet_conn_id} ${all_flows} Stop Mininet [Documentation] Stop Mininet. MininetKeywords.Stop Mininet And Exit Check No Switches [Documentation] Check no switches in topology. BuiltIn.Wait Until Keyword Succeeds 5s 1s ClusterOpenFlow.Check No Switches On Member ${SWITCHES} *** Keywords *** Initialization Phase [Documentation] Create controller session and set variables. SetupUtils.Setup_Utils_For_Setup_And_Teardown RequestsLibrary.Create Session session http://${ODL_SYSTEM_IP}:${RESTCONFPORT} auth=${AUTH} ${switches}= Convert To Integer ${SWITCHES} ${iter}= Convert To Integer ${ITER} ${all_groups}= BuiltIn.Evaluate ${switches} * ${iter} * 2 ${less_groups}= BuiltIn.Evaluate ${all_groups} - ${switches} * 2 # Stale flows/groups feature enabled in Boron onwards. ${all_flows}= BuiltIn.Evaluate ${switches} * ${iter+1} ${less_flows}= BuiltIn.Evaluate ${all_flows} - ${switches} # Stale flows/groups feature enabled in Boron onwards. BuiltIn.Set Suite Variable ${switches} BuiltIn.Set Suite Variable ${iter} BuiltIn.Set Suite Variable ${all_groups} BuiltIn.Set Suite Variable ${less_groups} BuiltIn.Set Suite Variable ${all_flows} BuiltIn.Set Suite Variable ${less_flows} BuiltIn.Set Suite Variable ${no_flows} ${SWITCHES} Final Phase [Documentation] Delete all sessions. ${command}= BuiltIn.Set Variable sudo iptables -v -F BuiltIn.Run Keyword And Ignore Error ClusterManagement.Run_Bash_Command_On_List_Or_All ${command} BuiltIn.Run Keyword And Ignore Error RequestsLibrary.DELETE On Session session url=${RFC8040_NODES_API} RequestsLibrary.Delete All Sessions Add Groups And Flows On Member [Documentation] Add ${ITER} groups type 1 & 2 and flows in every switch. [Arguments] ${iter}=1 ${member_index}=1 ${session}= Resolve_Http_Session_For_Member member_index=${member_index} FOR ${switch} IN RANGE 1 ${switches+1} &{mapping}= BuiltIn.Create_Dictionary NODE=openflow:${switch} TemplatedRequests.Post As Json Templated ... folder=${VAR_DIR}/add-group-1 ... mapping=${mapping} ... session=${session} ... iterations=${iter} TemplatedRequests.Post As Json Templated ... folder=${VAR_DIR}/add-group-2 ... mapping=${mapping} ... session=${session} ... iterations=${iter} TemplatedRequests.Post As Json Templated ... folder=${VAR_DIR}/add-flow ... mapping=${mapping} ... session=${session} ... iterations=${iter} END Add Single Group And Flow On Member [Documentation] Add 1 group 1&2 and 1 flow in every switch. [Arguments] ${member_index}=1 Add Groups And Flows On Member 1 ${member_index} Remove Single Group And Flow On Member [Documentation] Remove 1 group 1&2 and 1 flow in every switch. [Arguments] ${member_index}=1 ${session}= Resolve_Http_Session_For_Member member_index=${member_index} FOR ${switch} IN RANGE 1 ${switches+1} RequestsLibrary.DELETE On Session ... ${session} ... url=${RFC8040_NODES_API}/node=openflow%3A${switch}/flow-node-inventory:table=0/flow=1 RequestsLibrary.DELETE On Session ... ${session} ... url=${RFC8040_NODES_API}/node=openflow%3A${switch}/flow-node-inventory:group=1 RequestsLibrary.DELETE On Session ... ${session} ... url=${RFC8040_NODES_API}/node=openflow%3A${switch}/flow-node-inventory:group=1000 END Check Flow Stats Are Not Frozen [Documentation] Verify flow stats are not frozen for flow 1 and switch 1. [Arguments] ${member_index}=1 ${period_in_seconds}=5 ${duration_1}= Extract Flow Duration ${member_index} ${duration_1}= Builtin.Convert To Integer ${duration_1} BuiltIn.Sleep ${period_in_seconds} ${duration_2}= Extract Flow Duration ${member_index} ${duration_2}= Builtin.Convert To Integer ${duration_2} Should Not Be Equal As Integers ${duration_1} ${duration_2} Extract Flow Duration [Documentation] Extract duration for flow 1 in switch 1. [Arguments] ${member_index} ${session}= Resolve_Http_Session_For_Member member_index=${member_index} ${resp}= RequestsLibrary.GET On Session ... ${session} ... url=${RFC8040_NODES_API}/node=openflow%3A1/flow-node-inventory:table=0/flow=1?content=nonconfig ... headers=${headers} Log ${resp.content} ${json_resp}= Utils.Json Parse From String ${resp.content} ${flow_list}= Collections.Get_From_Dictionary ${json_resp} flow-node-inventory:flow ${flow_stats}= Collections.Get_From_Dictionary ... ${flow_list}[0] ... opendaylight-flow-statistics:flow-statistics ${duration}= Collections.Get_From_Dictionary ${flow_stats}[duration] second RETURN ${duration}