[Arguments] ${controller_index_list} ${device_type} ${device}
[Documentation] Checks Entity Owner status for a ${device} and returns owner index and list of candidates from a ${controller_index_list}.
... ${device_type} can be openflow, ovsdb, etc.
- ${length}= Get Length ${controller_index_list}
+ ${owner} ${candidates_list}= Get Device Entity Owner And Candidates Indexes controller@{controller_index_list}[0] ${device_type} ${device}
+ List Should Contain Value ${controller_index_list} ${owner} Owner ${owner} not exisiting in ${controller_index_list}
+ List Should Contain Sublist ${candidates_list} ${controller_index_list} Candidates are missing in ${candidates_list}
+ Remove Values From List ${candidates_list} ${owner}
+ [Return] ${owner} ${candidates_list}
+
+Get Device Entity Owner And Followers Indexes
+ [Arguments] ${session} ${device_type} ${device}
+ [Documentation] Returns the owner and followers indexes for a ${device}. Follower list = candidate list - owner
+ ${owner} ${candidates_list}= Get Device Entity Owner And Candidates Indexes ${session} ${device_type} ${device}
+ Remove Values From List ${candidates_list} ${owner}
+ [Return] ${owner} ${candidates_list}
+
+Get Device Entity Owner And Candidates Indexes
+ [Arguments] ${session} ${device_type} ${device}
+ [Documentation] Returns the owner and candidates indexes for a ${device}. Returns raw information, does not check missing
+ ... cluster nodes or so.
${candidates_list}= Create List
- ${data}= Utils.Get Data From URI controller@{controller_index_list}[0] /restconf/operational/entity-owners:entity-owners
+ ${data}= Utils.Get Data From URI ${session} /restconf/operational/entity-owners:entity-owners
Log ${data}
- ${clear_data}= Run Keyword If '${device_type}' == 'openflow' Extract OpenFlow Device Data ${data}
- ... ELSE IF '${device_type}' == 'ovsdb' Extract Ovsdb Device Data ${data}
- ... ELSE Fail Not recognized device type: ${device_type}
+ ${clear_data}= Run Keyword If '${device_type}' == 'openflow' Extract OpenFlow Device Data ${data} ELSE IF '${device_type}' == 'ovsdb'
+ ... Extract Ovsdb Device Data ${data} ELSE Fail Not recognized device type: ${device_type}
${json}= RequestsLibrary.To Json ${clear_data}
${entity_type_list}= Get From Dictionary &{json}[entity-owners] entity-type
${entity_type_index}= Get Index From List Of Dictionaries ${entity_type_list} type ${device_type}
Should Not Be Empty ${entity_owner} No owner found for ${device}
${owner}= Replace String ${entity_owner} member- ${EMPTY}
${owner}= Convert To Integer ${owner}
- List Should Contain Value ${controller_index_list} ${owner} Owner ${owner} not exisiting in ${controller_index_list}
${entity_candidates_list}= Get From Dictionary @{entity_list}[${entity_index}] candidate
- ${list_length}= Get Length ${entity_candidates_list}
: FOR ${entity_candidate} IN @{entity_candidates_list}
\ ${candidate}= Replace String &{entity_candidate}[name] member- ${EMPTY}
\ ${candidate}= Convert To Integer ${candidate}
\ Append To List ${candidates_list} ${candidate}
- List Should Contain Sublist ${candidates_list} ${controller_index_list} Candidates are missing in ${candidates_list}
- Remove Values From List ${candidates_list} ${owner}
[Return] ${owner} ${candidates_list}
Extract OpenFlow Device Data
: FOR ${ip} IN @{controllers}
\ Run Command On Remote System ${ip} ${cmd}
+Stop Controller Node And Verify
+ [Arguments] ${node}
+ [Documentation] Stops the given node
+ @{leader_list}= BuiltIn.Create List ${node}
+ Kill One Or More Controllers @{leader_list}
+ Controller Down Check ${node}
+
Kill One Or More Controllers
[Arguments] @{controllers}
[Documentation] Give this keyword a scalar or list of controllers to be stopped.
: FOR ${ip} IN @{controllers}
\ Run Command On Remote System ${ip} ${cmd}
+Start Controller Node And Verify
+ [Arguments] ${node} ${start_timeout}
+ [Documentation] Starts the given node
+ @{controllers}= BuiltIn.CreateList ${node}
+ Start One Or More Controllers @{controllers}
+ Wait For Controller Sync ${start_timeout} @{controllers}
+
Wait For Cluster Sync
[Arguments] ${timeout} @{controllers}
[Documentation] Waits for one or more clustered controllers to report Sync Status as true.
${value}= Get From Dictionary ${json} value
${OperSyncStatus}= Get From Dictionary ${value} SyncStatus
Log Operational Sync Status: ${OperSyncStatus}
- Run Keyword If ${OperSyncStatus} and ${ConfSyncStatus} Set Test Variable ${SyncStatus} ${TRUE}
- ... ELSE Set Test Variable ${SyncStatus} ${FALSE}
+ Run Keyword If ${OperSyncStatus} and ${ConfSyncStatus} Set Test Variable ${SyncStatus} ${TRUE} ELSE Set Test Variable
+ ... ${SyncStatus} ${FALSE}
[Return] ${SyncStatus}
Clean One Or More Journals
Modify IPTables
[Arguments] ${isolated controller} ${controller} ${rule type}
[Documentation] Adds a rule, usually inserting or deleting an entry between two controllers.
- ${base string} Set Variable sudo iptables ${rule type} OUTPUT -p all --source
+ ${base string} Set Variable sudo /sbin/iptables ${rule type} OUTPUT -p all --source
${cmd string} Catenate ${base string} ${isolated controller} --destination ${controller} -j DROP
Run Command On Remote System ${isolated controller} ${cmd string}
${cmd string} Catenate ${base string} ${controller} --destination ${isolated controller} -j DROP
Run Command On Remote System ${isolated controller} ${cmd string}
- ${cmd string} Set Variable sudo iptables -L -n
+ ${cmd string} Set Variable sudo /sbin/iptables -L -n
${return string}= Run Command On Remote System ${isolated controller} ${cmd string}
#If inserting rules:
Run Keyword If "${rule type}" == '-I' Should Match Regexp ${return string} [\s\S]*DROP *all *-- *${isolated controller} *${controller}[\s\S]*
[Arguments] ${isolated controller}
[Documentation] This keyword is generally not called from a test case but supports a complete wipe of all rules on
... all contollers.
- ${cmd string} Set Variable sudo iptables -v -F
+ ${cmd string} Set Variable sudo /sbin/iptables -v -F
${return string}= Run Command On Remote System ${isolated controller} ${cmd string}
Log return: ${return string}
Should Contain ${return string} Flushing chain `INPUT'
--- /dev/null
+*** Settings ***
+Documentation Library to provide ovsdb access to mininet topologies
+Library SSHLibrary
+Library ${CURDIR}/VsctlListParser.py
+Library Collections
+
+*** Variables ***
+${SH_BR_CMD} ovs-vsctl list Bridge
+${SH_CNTL_CMD} ovs-vsctl list Controller
+${ovs_switch_data} ${None}
+${lprompt} mininet>
+${lcmd_prefix} sh
+
+*** Keywords ***
+Initialize If Shell Used
+ [Arguments] ${prompt} ${cmd_prefix}
+ BuiltIn.Set Suite variable ${lprompt} ${prompt}
+ BuiltIn.Set Suite variable ${lcmd_prefix} ${cmd_prefix}
+
+Get Ovsdb Data
+ [Arguments] ${prompt}=mininet>
+ [Documentation] Gets ovs data and parse them.
+ SSHLibrary.Write ${lcmd_prefix} ${SH_BR_CMD}
+ ${brstdout}= SSHLibrary.Read_Until ${lprompt}
+ Log ${brstdout}
+ SSHLibrary.Write ${lcmd_prefix} ${SH_CNTL_CMD}
+ ${cntlstdout}= SSHLibrary.Read_Until ${lprompt}
+ Log ${cntlstdout}
+ ${data} ${bridegs} ${controllers}= VsctlListParser.Parse ${brstdout} ${cntlstdout}
+ BuiltIn.Log ${data}
+ BuiltIn.Set Suite Variable ${ovs_switch_data} ${data}
+ BuiltIn.Return From Keyword ${data}
+
+Get Controllers Uuid
+ [Arguments] ${switch} ${controller} ${update_data}=${False}
+ [Documentation] Returns controllers uuid
+ Run Keyword If ${update_data}==${True} Get Ovsdb Data
+ ${bridge}= Collections.Get From Dictionary ${ovs_switch_data} ${switch}
+ ${cntls}= Collections.Get From Dictionary ${bridge} controller
+ ${cntl}= Collections.Get From Dictionary ${cntls} ${controller}
+ ${uuid}= Collections.Get From Dictionary ${cntl} _uuid
+ BuiltIn.Return From Keyword ${uuid}
+
+Execute OvsVsctl Show Command
+ [Documentation] Executes ovs-vsctl show command and returns stdout, no check nor change is performed
+ SSHLibrary.Write ${lcmd_prefix} ovs-vsctl show
+ ${output}= SSHLibrary.Read_Until ${lprompt}
+ Log ${output}
+
+Set Bridge Controllers
+ [Arguments] ${bridge} ${controllers} ${disconnected}=${False}
+ [Documentation] Adds controller to the bridge
+ ${cmd}= BuiltIn.Set Variable ${lcmd_prefix} ovs-vsctl set-controller ${bridge}
+ : FOR ${cntl} IN @{controllers}
+ \ ${cmd}= BuiltIn.Set Variable If ${disconnected}==${False} ${cmd} tcp:${cntl}:6653 ${cmd} tcp:${cntl}:6654
+ BuiltIn.Log ${cmd}
+ SSHLibrary.Write ${cmd}
+ ${output}= SSHLibrary.Read_Until ${lprompt}
+ Log ${output}
+
+Disconnect Switch From Controller And Verify Disconnected
+ [Arguments] ${switch} ${controller} ${update_data}=${False} ${verify_disconnected}=${True}
+ [Documentation] Disconnects the switch from the controller by setting the incorrect port
+ Run Keyword If ${update_data}==${True} Get Ovsdb Data
+ ${uuid}= Get Controllers Uuid ${switch} ${controller}
+ ${cmd}= BuiltIn.Set Variable ${lcmd_prefix} ovs-vsctl set Controller ${uuid} target="tcp\\:${controller}\\:6654"
+ SSHLibrary.Write ${cmd}
+ ${output}= SSHLibrary.Read_Until ${lprompt}
+ Log ${output}
+ Return From Keyword If ${verify_disconnected}==${False}
+ BuiltIn.Wait Until Keyword Succeeds 5x 2s Should Be Disconnected ${switch} ${controller} update_data=${True}
+ [Teardown] Execute OvsVsctl Show Command
+
+Reconnect Switch To Controller And Verify Connected
+ [Arguments] ${switch} ${controller} ${update_data}=${False} ${verify_connected}=${True}
+ [Documentation] Reconnects the switch back to the controller by setting the correct port
+ Run Keyword If ${update_data}==${True} Get Ovsdb Data
+ ${uuid}= Get Controllers Uuid ${switch} ${controller}
+ ${cmd}= BuiltIn.Set Variable ${lcmd_prefix} ovs-vsctl set Controller ${uuid} target="tcp\\:${controller}\\:6653"
+ SSHLibrary.Write ${cmd}
+ ${output}= SSHLibrary.Read_Until ${lprompt}
+ Log ${output}
+ Return From Keyword If ${verify_connected}==${False}
+ BuiltIn.Wait Until Keyword Succeeds 5x 2s Should Be Connected ${switch} ${controller} update_data=${True}
+ [Teardown] Execute OvsVsctl Show Command
+
+Should Be Connected
+ [Arguments] ${switch} ${controller} ${update_data}=${False}
+ [Documentation] Check if the switch is connected
+ ${connected}= OvsManager__Is_Connected ${switch} ${controller} update_data=${update_data}
+ BuiltIn.Should Be True ${connected}
+
+Should Be Disconnected
+ [Arguments] ${switch} ${controller} ${update_data}=${False}
+ [Documentation] Check if the switch is disconnected
+ ${connected}= OvsManager__Is_Connected ${switch} ${controller} update_data=${update_data}
+ BuiltIn.Should Be Equal ${connected} ${False}
+
+OvsManager__Is_Connected
+ [Arguments] ${switch} ${controller} ${update_data}=${False}
+ [Documentation] Return is_connected boolean value
+ Run Keyword If ${update_data}==${True} Get Ovsdb Data
+ ${bridge}= Collections.Get From Dictionary ${ovs_switch_data} ${switch}
+ ${cntls}= Collections.Get From Dictionary ${bridge} controller
+ ${cntl}= Collections.Get From Dictionary ${cntls} ${controller}
+ ${connected}= Collections.Get From Dictionary ${cntl} is_connected
+ [Teardown] Execute OvsVsctl Show Command
+ [Return] ${connected}
+
+Should Be Master
+ [Arguments] ${switch} ${controller} ${update_data}=${False}
+ [Documentation] Verifies the master role
+ ${role} Get Node Role ${switch} ${controller} update_data=${update_data}
+ BuiltIn.Should Be Equal ${role} master
+
+Should Be Slave
+ [Arguments] ${switch} ${controller} ${update_data}=${False}
+ [Documentation] Verifies the slave role
+ ${role} Get Node Role ${switch} ${controller} update_data=${update_data}
+ BuiltIn.Should Be Equal ${role} slave
+
+Get Node Role
+ [Arguments] ${switch} ${controller} ${update_data}=${False}
+ [Documentation] Returns the controllers role
+ Run Keyword If ${update_data}==${True} Get Ovsdb Data
+ ${bridge}= Collections.Get From Dictionary ${ovs_switch_data} ${switch}
+ ${cntls}= Collections.Get From Dictionary ${bridge} controller
+ ${cntl}= Collections.Get From Dictionary ${cntls} ${controller}
+ ${role}= Collections.Get From Dictionary ${cntl} role
+ Return From Keyword ${role}
+
+Get Master Node
+ [Arguments] ${switch} ${update_data}=${False}
+ [Documentation] Gets controller which is a master
+ ${master}= BuiltIn.Set Variable ${None}
+ Run Keyword If ${update_data}==${True} Get Ovsdb Data
+ ${bridge}= Collections.Get From Dictionary ${ovs_switch_data} ${switch}
+ ${cntls_dict}= Collections.Get From Dictionary ${bridge} controller
+ ${cntls_items}= Collections.Get Dictionary Items ${cntls_dict}
+ : FOR ${key} ${value} IN @{cntls_items}
+ \ Log ${key} : ${value}
+ \ ${role}= Collections.Get From Dictionary ${value} role
+ \ Run Keyword If "${role}"=="master" BuiltIn.Should Be Equal ${master} ${None}
+ \ ${master}= BuiltIn.Set Variable If "${role}"=="master" ${key} ${master}
+ BuiltIn.Should Not Be Equal ${master} ${None}
+ Return From Keyword ${master}
+
+Get Slave Nodes
+ [Arguments] ${switch} ${update_data}=${False}
+ [Documentation] Returns a list of ips of slave nodes for particular switch
+ ${slaves}= BuiltIn.Create List
+ Run Keyword If ${update_data}==${True} Get Ovsdb Data
+ ${bridge}= Collections.Get From Dictionary ${ovs_switch_data} ${switch}
+ ${cntls_dict}= Collections.Get From Dictionary ${bridge} controller
+ ${cntls_items}= Collections.Get Dictionary Items ${cntls_dict}
+ : FOR ${key} ${value} IN @{cntls_items}
+ \ Log ${key} : ${value}
+ \ ${role}= Collections.Get From Dictionary ${value} role
+ \ Run Keyword If "${role}"=="slave" Collections.Append To List ${slaves} ${key}
+ Return From Keyword ${slaves}
+
+Setup Clustered Controller For Switches
+ [Arguments] ${switches} ${controller_ips} ${verify_connected}=${False}
+ [Documentation] The idea of this keyword is to setup clustered controller and to be more or less sure that the role is filled correctly. The problem is when
+ ... more controllers are being set up at once, the role shown in Controller ovsdb table is not the same as we can see from wireshark traces.
+ ... Now we set disconnected controllers and we will connect them expecting that the first connected controller will be master.
+ : FOR ${switch_name} IN @{switches}
+ \ Set Bridge Controllers ${switch_name} ${controller_ips} disconnected=${True}
+ # now we need to enable one node which will be master
+ OvsManager.Get Ovsdb Data
+ : FOR ${switch_name} IN @{switches}
+ \ ${own}= Collections.Get From List ${controller_ips} 0
+ \ Reconnect Switch To Controller And Verify Connected ${switch_name} ${own} verify_connected=${False}
+ # now we need to wait till master controllers are connected
+ BuiltIn.Wait Until Keyword Succeeds 5x 2s OvsManager__Verify_Masters_Connected ${switches} update_data=${True}
+ # now we can enable slaves
+ OvsManager__Enable_Slaves ${switches} verify_connected=${verify_connected}
+
+OvsManager__Verify_Masters_Connected
+ [Arguments] ${switches} ${update_data}=${False}
+ [Documentation] Private keyword, the existence of master means it is verified
+ Run Keyword If ${update_data}==${True} Get Ovsdb Data
+ : FOR ${switch_name} IN @{switches}
+ \ Get Master Node ${switch_name}
+
+OvsManager__Enable_Slaves
+ [Arguments] ${switches} ${update_data}=${False} ${verify_connected}=${False}
+ [Documentation] This is a private keyword to enable diconnected controllers
+ Run Keyword If ${update_data}==${True} Get Ovsdb Data
+ : FOR ${switch_name} IN @{switches}
+ \ OvsManager__Enable_Slaves_For_Switch ${switch_name} verify_connected=${verify_connected}
+
+OvsManager__Enable_Slaves_For_Switch
+ [Arguments] ${switch} ${update_data}=${False} ${verify_connected}=${False}
+ [Documentation] This is a private keyword, verification is not reliable yet, enables disconnected controllers
+ Run Keyword If ${update_data}==${True} Get Ovsdb Data
+ ${bridge}= Collections.Get From Dictionary ${ovs_switch_data} ${switch}
+ ${cntls_dict}= Collections.Get From Dictionary ${bridge} controller
+ ${cntls_items}= Collections.Get Dictionary Items ${cntls_dict}
+ : FOR ${cntl_id} ${cntl_value} IN @{cntls_items}
+ \ Log ${cntl_id} : ${cntl_value}
+ \ ${role}= Collections.Get From Dictionary ${cntl_value} role
+ \ ${connected}= Collections.Get From Dictionary ${cntl_value} is_connected
+ \ Run Keyword If ${connected}==${False} Reconnect Switch To Controller And Verify Connected ${switch} ${cntl_id} verify_connected=${verify_connected}
--- /dev/null
+"""
+Library for the robot based system test tool of the OpenDaylight project.
+
+This library will parse 'ovs-vstcl list Bridge' and 'ovs-vstcl list Controller'
+commands and create dictionaries with parsed details which will be used
+for another pruposes.
+
+Authors: pgubka@cisco.com
+Created: 2016-04-04
+"""
+
+import re
+import copy
+import sys
+import traceback
+
+
+def _parse_stdout(stdout):
+ """ Transforms stdout to dict """
+ text = stdout.replace(" ", "")
+ text = text.replace("\r", "")
+ pat = re.compile(r'(?P<key>\w+):(?P<value>.+)')
+ regroups = re.finditer(pat, text)
+ outdict = {}
+ for g in regroups:
+ print g.group()
+ if g.group('key') == '_uuid':
+ cntl_uuid = g.group('value')
+ outdict[cntl_uuid] = {}
+ outdict[cntl_uuid][g.group('key')] = g.group('value')
+ return outdict
+
+
+def _postprocess_data(bridges, controllers):
+ """What is done here:
+ - merge bridges and controllers
+ - replace controller 'key' (ip instead uuid)
+ - transformed controller's connected status to bool
+ """
+ brs = copy.deepcopy(bridges)
+ cntls = copy.deepcopy(controllers)
+
+ # replacing string value of is_connected key with boolean
+ for key, cntl in cntls.iteritems():
+ if cntl['is_connected'] == 'false':
+ cntl['is_connected'] = False
+ elif cntl['is_connected'] == 'true':
+ cntl['is_connected'] = True
+ else:
+ cntl['is_connected'] = None
+
+ # replacing keys with the same values
+ for key, value in bridges.iteritems():
+ brs[value['name'][1:-1]] = brs[key]
+ del brs[key]
+
+ for key, value in brs.iteritems():
+ # replace string with references with dict of controllers
+ ctl_refs = value['controller'][1:-1].split(',')
+ value['controller'] = {}
+ for ctl_ref in ctl_refs:
+ if ctl_ref is not '':
+ value['controller'][ctl_ref] = cntls[ctl_ref]
+
+ for brkey, bridge in brs.iteritems():
+ new_cntls = {}
+ for cnkey, cntl in bridge['controller'].iteritems():
+ # port 6654 is set by OvsMAnager.robot to disconnect from controller
+ if '6653' in cntl['target'] or '6633' in cntl['target'] or '6654' in cntl['target']:
+ new_key = cntl['target'].split(":")[1] # getting middle from "tcp:ip:6653"
+ else:
+ new_key = cntl['target'][1:-1] # getting string without quotes "ptcp:6638"
+ new_cntls[new_key] = cntl
+ bridge['controller'] = new_cntls
+
+ return brs
+
+
+def parse(bridge_stdout, cntl_stdout):
+ """Produces dictionary with data for future usege
+
+ Args:
+ :param bridge_stdout: output of 'ovs-vsclt list Bridge' command
+
+ :param cntl_stdout: output of 'ovs-vsclt list Controller' command
+
+ Returns:
+ :returns processed: processed output dictionary
+
+ :returns bridges: list bridge command output transformed to dist
+
+ :returns controllers: list controller command output transformed to dist
+ """
+
+ try:
+ bridges = _parse_stdout(bridge_stdout)
+ controllers = _parse_stdout(cntl_stdout)
+ processed = _postprocess_data(bridges, controllers)
+ except Exception:
+ traceback.print_exc()
+ raise
+ return processed, bridges, controllers
--- /dev/null
+*** Settings ***
+Documentation Test suite for entity ownership service and openflowplugin. Makes changes on switch side.
+Suite Setup Start Suite
+Suite Teardown End Suite
+Test Template Reconnecting Switch Scenario
+Library SSHLibrary
+Library RequestsLibrary
+Library XML
+Resource ${CURDIR}/../../../libraries/Utils.robot
+Resource ${CURDIR}/../../../libraries/OvsManager.robot
+Resource ${CURDIR}/../../../libraries/ClusterKeywords.robot
+Library Collections
+
+*** Variables ***
+${SWITCHES} 1
+# this is for mininet 2.2.1 ${START_CMD} sudo mn --controller=remote,ip=${ODL_SYSTEM_1_IP} --controller=remote,ip=${ODL_SYSTEM_2_IP} --controller=remote,ip=${ODL_SYSTEM_3_IP} --topo linear,${SWITCHES} --switch ovsk,protocols=OpenFlow13
+${START_CMD} sudo mn --topo linear,${SWITCHES} --switch ovsk,protocols=OpenFlow13
+@{cntls_idx_list} ${1} ${2} ${3}
+
+*** Test Cases ***
+Switches To Be Connected To All Nodes
+ [Documentation] Initial check for correct connected topology.
+ [Template] NONE
+ BuiltIn.Wait Until Keyword Succeeds 5x 3s Check All Switches Connected To All Cluster Nodes
+
+Reconnecting Switch s1
+ s1
+
+Switches Still Be Connected To All Nodes
+ [Template] NONE
+ BuiltIn.Wait Until Keyword Succeeds 5x 3s Check All Switches Connected To All Cluster Nodes
+
+*** Keywords ***
+Start Suite
+ BuiltIn.Log Start the test on the base edition
+ ${mininet_conn_id}= SSHLibrary.Open Connection ${TOOLS_SYSTEM_IP} prompt=${TOOLS_SYSTEM_PROMPT}
+ BuiltIn.Set Suite Variable ${mininet_conn_id}
+ SSHLibrary.Login With Public Key ${TOOLS_SYSTEM_USER} ${USER_HOME}/.ssh/id_rsa any
+ SSHLibrary.Put File ${CURDIR}/../../../libraries/DynamicMininet.py .
+ SSHLibrary.Execute Command sudo ovs-vsctl set-manager ptcp:6644
+ SSHLibrary.Execute Command sudo mn -c
+ SSHLibrary.Write ${START_CMD}
+ SSHLibrary.Read Until mininet>
+ ${cntls_list} BuiltIn.Create List ${ODL_SYSTEM_1_IP} ${ODL_SYSTEM_2_IP} ${ODL_SYSTEM_3_IP}
+ ${switch_list} BuiltIn.Create List
+ : FOR ${i} IN RANGE 0 ${SWITCHES}
+ \ ${sid}= BuiltIn.Evaluate ${i}+1
+ \ Collections.Append To List ${switch_list} s${sid}
+ : FOR ${i} IN RANGE 0 ${NUM_ODL_SYSTEM}
+ \ ${cid}= BuiltIn.Evaluate ${i}+1
+ \ RequestsLibrary.Create Session controller${cid} http://${ODL_SYSTEM_${cid}_IP}:${RESTCONFPORT} auth=${AUTH} headers=${HEADERS_XML}
+ BuiltIn.Set Suite Variable ${active_session} controller1
+ OvsManager.Setup Clustered Controller For Switches ${switch_list} ${cntls_list}
+ BuiltIn.Wait Until Keyword Succeeds 10s 1s Are Switches Connected Topo
+
+End Suite
+ RequestsLibrary.Delete All Sessions
+ Utils.Stop Suite
+
+Are Switches Connected Topo
+ [Documentation] Checks wheather switches are connected to controller
+ ${resp}= RequestsLibrary.Get Request ${active_session} ${OPERATIONAL_TOPO_API}/topology/flow:1 headers=${ACCEPT_XML}
+ BuiltIn.Log ${resp.content}
+ ${count}= XML.Get Element Count ${resp.content} xpath=node
+ BuiltIn.Should Be Equal As Numbers ${count} ${SWITCHES}
+
+Check All Switches Connected To All Cluster Nodes
+ [Documentation] Verifies all switches are connected to all cluster nodes
+ OvsManager.Get Ovsdb Data
+ : FOR ${i} IN RANGE 0 ${SWITCHES}
+ \ ${sid}= BuiltIn.Evaluate ${i}+1
+ \ OvsManager.Should Be Connected s${sid} ${ODL_SYSTEM_1_IP} update_data=${False}
+ \ OvsManager.Should Be Connected s${sid} ${ODL_SYSTEM_2_IP} update_data=${False}
+ \ OvsManager.Should Be Connected s${sid} ${ODL_SYSTEM_3_IP} update_data=${False}
+
+Reconnecting Switch Scenario
+ [Arguments] ${switch_name}
+ [Documentation] Disconnect and connect master and follower and check switch data to be consistent
+ ${idx}= BuiltIn.Evaluate str("${switch_name}"[1:])
+ BuiltIn.Set Test Variable ${idx}
+ Disconnect Switchs Old Master ${switch_name}
+ Reconnect Switchs Old Master ${switch_name}
+ Disconnect Switchs Follower ${switch_name}
+ Reconnect Switchs Follower ${switch_name}
+ [Teardown] Run Keyword If "${disc_cntl}"!="${Empty}" OvsManager.Reconnect Switch To Controller And Verify Connected ${switch_name} ${disc_cntl}
+
+Disconnect Switchs Old Master
+ [Arguments] ${switch_name}
+ BuiltIn.Set Test Variable ${disc_cntl} ${Empty}
+ Check Count Integrity ${switch_name} expected_controllers=3
+ ${old_owner} ${old_followers}= ClusterKeywords.Get Device Entity Owner And Followers Indexes ${active_session} openflow openflow:${idx}
+ ${old_master}= BuiltIn.Set Variable ${ODL_SYSTEM_${old_owner}_IP}
+ OvsManager.Disconnect Switch From Controller And Verify Disconnected ${switch_name} ${old_master}
+ BuiltIn.Set Test Variable ${disc_cntl} ${old_master}
+ ${new_master}= BuiltIn.Wait Until Keyword Succeeds 5x 3s Verify New Master Controller Node ${switch_name} ${old_master}
+ ${owner} ${followers}= ClusterKeywords.Get Device Entity Owner And Followers Indexes ${active_session} openflow openflow:${idx}
+ Collections.List Should Contain Value ${old_followers} ${owner}
+ Check Count Integrity ${switch_name} expected_controllers=2
+ BuiltIn.Should Be Equal As Strings ${new_master} ${ODL_SYSTEM_${owner}_IP}
+ BuiltIn.Set Test Variable ${old_owner}
+ BuiltIn.Set Test Variable ${old_followers}
+ BuiltIn.Set Test Variable ${old_master}
+ BuiltIn.Set Test Variable ${owner}
+ BuiltIn.Set Test Variable ${new_master}
+
+Reconnect Switchs Old Master
+ [Arguments] ${switch_name}
+ OvsManager.Reconnect Switch To Controller And Verify Connected ${switch_name} ${old_master}
+ BuiltIn.Set Test Variable ${disc_cntl} ${Empty}
+ BuiltIn.Wait Until Keyword Succeeds 5x 3s Verify Follower Added ${switch_name} ${old_owner}
+ ${new_owner} ${new_followers}= ClusterKeywords.Get Device Entity Owner And Followers Indexes ${active_session} openflow openflow:${idx}
+ Check Count Integrity ${switch_name} expected_controllers=3
+ BuiltIn.Should Be Equal ${owner} ${new_owner}
+ Collections.List Should Contain Value ${new_followers} ${old_owner}
+
+Disconnect Switchs Follower
+ [Arguments] ${switch_name}
+ ${old_owner} ${old_followers}= ClusterKeywords.Get Device Entity Owner And Followers Indexes ${active_session} openflow openflow:${idx}
+ ${old_follower}= Collections.Get From List ${old_followers} 0
+ ${old_slave}= BuiltIn.Set Variable ${ODL_SYSTEM_${old_follower}_IP}
+ OvsManager.Disconnect Switch From Controller And Verify Disconnected ${switch_name} ${old_slave}
+ BuiltIn.Set Test Variable ${disc_cntl} ${old_slave}
+ BuiltIn.Wait Until Keyword Succeeds 5x 3s Check Count Integrity ${switch_name} expected_controllers=2
+ ${owner} ${followers}= ClusterKeywords.Get Device Entity Owner And Followers Indexes ${active_session} openflow openflow:${idx}
+ BuiltIn.Should Be Equal ${owner} ${old_owner}
+ Collections.List Should Not Contain Value ${followers} ${old_follower}
+ BuiltIn.Should Be Equal As Strings ${new_master} ${ODL_SYSTEM_${owner}_IP}
+ BuiltIn.Set Test Variable ${old_owner}
+ BuiltIn.Set Test Variable ${old_followers}
+ BuiltIn.Set Test Variable ${old_follower}
+ BuiltIn.Set Test Variable ${old_slave}
+ BuiltIn.Set Test Variable ${owner}
+
+Reconnect Switchs Follower
+ [Arguments] ${switch_name}
+ OvsManager.Reconnect Switch To Controller And Verify Connected ${switch_name} ${old_slave}
+ BuiltIn.Set Test Variable ${disc_cntl} ${Empty}
+ BuiltIn.Wait Until Keyword Succeeds 5x 3s Check Count Integrity ${switch_name} expected_controllers=3
+ ${new_owner} ${new_followers}= ClusterKeywords.Get Device Entity Owner And Followers Indexes ${active_session} openflow openflow:${idx}
+ BuiltIn.Should Be Equal ${old_owner} ${new_owner}
+ Collections.List Should Contain Value ${new_followers} ${old_follower}
+
+Check Count Integrity
+ [Arguments] ${switch_name} ${expected_controllers}=3
+ [Documentation] Every switch must have only one master and rest must be followers and together must be of expected nodes count
+ ${idx}= BuiltIn.Evaluate "${switch_name}"[1:]
+ ${owner} ${candidates}= ClusterKeywords.Get Device Entity Owner And Candidates Indexes ${active_session} openflow openflow:${idx}
+ ${count}= BuiltIn.Get Length ${candidates}
+ BuiltIn.Should Be Equal As Numbers ${expected_controllers} ${count}
+
+Verify New Master Controller Node
+ [Arguments] ${switch_name} ${old_master}
+ [Documentation] Checks if given node is different from actual master
+ ${idx}= BuiltIn.Evaluate "${switch_name}"[1:]
+ ${owner} ${followers}= ClusterKeywords.Get Device Entity Owner And Followers Indexes ${active_session} openflow openflow:${idx}
+ ${new_master} BuiltIn.Set Variable ${ODL_SYSTEM_${owner}_IP}
+ BuiltIn.Should Not Be Equal ${old_master} ${new_master}
+ Return From Keyword ${new_master}
+
+Verify Follower Added
+ [Arguments] ${switch_name} ${expected_node}
+ [Documentation] Checks if given node is in the list of active followers
+ ${idx}= BuiltIn.Evaluate "${switch_name}"[1:]
+ ${owner} ${followers}= ClusterKeywords.Get Device Entity Owner And Followers Indexes ${active_session} openflow openflow:${idx}
+ Collections.List Should Contain Value ${followers} ${expected_node}
--- /dev/null
+*** Settings ***
+Documentation Test suite for entity ownership service and openflowplugin. Makes changes on controller side (isolating cluster node)
+Suite Setup Start Suite
+Suite Teardown End Suite
+Test Template Isolating Node Scenario
+Library SSHLibrary
+Library RequestsLibrary
+Library XML
+Resource ${CURDIR}/../../../libraries/Utils.robot
+Resource ${CURDIR}/../../../libraries/OvsManager.robot
+Resource ${CURDIR}/../../../libraries/ClusterKeywords.robot
+Library Collections
+
+*** Variables ***
+${SWITCHES} 1
+# this is for mininet 2.2.1 ${START_CMD} sudo mn --controller=remote,ip=${ODL_SYSTEM_1_IP} --controller=remote,ip=${ODL_SYSTEM_2_IP} --controller=remote,ip=${ODL_SYSTEM_3_IP} --topo linear,${SWITCHES} --switch ovsk,protocols=OpenFlow13
+${START_CMD} sudo mn --topo linear,${SWITCHES} --switch ovsk,protocols=OpenFlow13
+@{CONTROLLER_NODES} ${ODL_SYSTEM_1_IP} ${ODL_SYSTEM_2_IP} ${ODL_SYSTEM_3_IP}
+@{cntls_idx_list} ${1} ${2} ${3}
+
+*** Test Cases ***
+Switches To Be Connected To All Nodes
+ [Documentation] Initial check for correct connected topology.
+ [Template] NONE
+ BuiltIn.Wait Until Keyword Succeeds 5x 3s Check All Switches Connected To All Cluster Nodes
+
+Isolating Owner Of Switch s1
+ s1
+
+Switches Still Be Connected To All Nodes
+ [Template] NONE
+ BuiltIn.Wait Until Keyword Succeeds 5x 3s Check All Switches Connected To All Cluster Nodes
+
+*** Keywords ***
+Start Suite
+ BuiltIn.Log Start the test on the base edition
+ ${mininet_conn_id}= SSHLibrary.Open Connection ${TOOLS_SYSTEM_IP} prompt=${TOOLS_SYSTEM_PROMPT}
+ BuiltIn.Set Suite Variable ${mininet_conn_id}
+ SSHLibrary.Login With Public Key ${TOOLS_SYSTEM_USER} ${USER_HOME}/.ssh/id_rsa any
+ SSHLibrary.Put File ${CURDIR}/../../../libraries/DynamicMininet.py .
+ SSHLibrary.Execute Command sudo ovs-vsctl set-manager ptcp:6644
+ SSHLibrary.Execute Command sudo mn -c
+ SSHLibrary.Write ${START_CMD}
+ SSHLibrary.Read Until mininet>
+ ${cntls_list} BuiltIn.Create List ${ODL_SYSTEM_1_IP} ${ODL_SYSTEM_2_IP} ${ODL_SYSTEM_3_IP}
+ ${switch_list} BuiltIn.Create List
+ : FOR ${i} IN RANGE 0 ${SWITCHES}
+ \ ${sid}= BuiltIn.Evaluate ${i}+1
+ \ Collections.Append To List ${switch_list} s${sid}
+ : FOR ${i} IN RANGE 0 ${NUM_ODL_SYSTEM}
+ \ ${cid}= BuiltIn.Evaluate ${i}+1
+ \ RequestsLibrary.Create Session controller${cid} http://${ODL_SYSTEM_${cid}_IP}:${RESTCONFPORT} auth=${AUTH} headers=${HEADERS_XML}
+ BuiltIn.Set Suite Variable ${active_session} controller1
+ OvsManager.Setup Clustered Controller For Switches ${switch_list} ${cntls_list}
+ BuiltIn.Wait Until Keyword Succeeds 10s 1s Are Switches Connected Topo
+
+End Suite
+ RequestsLibrary.Delete All Sessions
+ Utils.Stop Suite
+
+Are Switches Connected Topo
+ [Documentation] Checks wheather switches are connected to controller
+ ${resp}= RequestsLibrary.Get Request ${active_session} ${OPERATIONAL_TOPO_API}/topology/flow:1 headers=${ACCEPT_XML}
+ BuiltIn.Log ${resp.content}
+ ${count}= XML.Get Element Count ${resp.content} xpath=node
+ BuiltIn.Should Be Equal As Numbers ${count} ${SWITCHES}
+
+Check All Switches Connected To All Cluster Nodes
+ [Documentation] Verifies all switches are connected to all cluster nodes
+ OvsManager.Get Ovsdb Data
+ : FOR ${i} IN RANGE 0 ${SWITCHES}
+ \ ${sid}= BuiltIn.Evaluate ${i}+1
+ \ OvsManager.Should Be Connected s${sid} ${ODL_SYSTEM_1_IP} update_data=${False}
+ \ OvsManager.Should Be Connected s${sid} ${ODL_SYSTEM_2_IP} update_data=${False}
+ \ OvsManager.Should Be Connected s${sid} ${ODL_SYSTEM_3_IP} update_data=${False}
+
+Isolating Node Scenario
+ [Arguments] ${switch_name}
+ [Documentation] Disconnect and connect master and follower and check switch data to be consistent
+ ${idx}= BuiltIn.Evaluate str("${switch_name}"[1:])
+ BuiltIn.Set Test Variable ${idx}
+ Isolate Switchs Old Owner ${switch_name}
+ Rejoin Switchs Old Owner ${switch_name}
+ Isolate Switchs Candidate ${switch_name}
+ Rejoin Switchs Candidate ${switch_name}
+ [Teardown] Run Keyword If "${isol_node}"!="${Empty}" Rejoin Controller To The Cluster ${isol_node}
+
+Isolate Switchs Old Owner
+ [Arguments] ${switch_name}
+ BuiltIn.Set Test Variable ${isol_node} ${Empty}
+ Check Count Integrity ${switch_name} expected_controllers=3
+ ${old_owner} ${old_followers}= ClusterKeywords.Get Device Entity Owner And Followers Indexes ${active_session} openflow openflow:${idx}
+ ${old_master}= BuiltIn.Set Variable ${ODL_SYSTEM_${old_owner}_IP}
+ ${tmp_follower}= Collections.Get From List ${old_followers} 0
+ BuiltIn.Set Suite Variable ${active_session} controller${tmp_follower}
+ Isolate Controller From The Cluster ${old_master}
+ BuiltIn.Set Test Variable ${isol_node} ${old_master}
+ ${new_master}= BuiltIn.Wait Until Keyword Succeeds 5x 3s Verify New Master Controller Node ${switch_name} ${old_master}
+ ${owner} ${followers}= ClusterKeywords.Get Device Entity Owner And Followers Indexes ${active_session} openflow openflow:${idx}
+ Collections.List Should Contain Value ${old_followers} ${owner}
+ Check Count Integrity ${switch_name} expected_controllers=2
+ BuiltIn.Should Be Equal As Strings ${new_master} ${ODL_SYSTEM_${owner}_IP}
+ BuiltIn.Set Suite Variable ${active_session} controller${owner}
+ BuiltIn.Set Test Variable ${old_owner}
+ BuiltIn.Set Test Variable ${old_followers}
+ BuiltIn.Set Test Variable ${old_master}
+ BuiltIn.Set Test Variable ${owner}
+ BuiltIn.Set Test Variable ${new_master}
+
+Rejoin Switchs Old Owner
+ [Arguments] ${switch_name}
+ Rejoin Controller To The Cluster ${old_master}
+ BuiltIn.Set Test Variable ${isol_node} ${Empty}
+ BuiltIn.Wait Until Keyword Succeeds 50x 3s Verify Follower Added ${switch_name} ${old_owner}
+ ${new_owner} ${new_followers}= ClusterKeywords.Get Device Entity Owner And Followers Indexes ${active_session} openflow openflow:${idx}
+ Check Count Integrity ${switch_name} expected_controllers=3
+ BuiltIn.Should Be Equal ${owner} ${new_owner}
+ Collections.List Should Contain Value ${new_followers} ${old_owner}
+
+Isolate Switchs Candidate
+ [Arguments] ${switch_name}
+ ${old_owner} ${old_followers}= ClusterKeywords.Get Device Entity Owner And Followers Indexes ${active_session} openflow openflow:${idx}
+ ${old_follower}= Collections.Get From List ${old_followers} 0
+ ${old_slave}= BuiltIn.Set Variable ${ODL_SYSTEM_${old_follower}_IP}
+ Isolate Controller From The Cluster ${old_slave}
+ BuiltIn.Set Test Variable ${isol_cntl} ${old_slave}
+ BuiltIn.Wait Until Keyword Succeeds 5x 3s Check Count Integrity ${switch_name} expected_controllers=2
+ ${owner} ${followers}= ClusterKeywords.Get Device Entity Owner And Followers Indexes ${active_session} openflow openflow:${idx}
+ BuiltIn.Should Be Equal ${owner} ${old_owner}
+ Collections.List Should Not Contain Value ${followers} ${old_follower}
+ BuiltIn.Should Be Equal As Strings ${new_master} ${ODL_SYSTEM_${owner}_IP}
+ BuiltIn.Set Test Variable ${old_owner}
+ BuiltIn.Set Test Variable ${old_followers}
+ BuiltIn.Set Test Variable ${old_follower}
+ BuiltIn.Set Test Variable ${old_slave}
+ BuiltIn.Set Test Variable ${owner}
+
+Rejoin Switchs Candidate
+ [Arguments] ${switch_name}
+ Rejoin Controller To The Cluster ${old_slave}
+ BuiltIn.Set Test Variable ${isol_node} ${Empty}
+ BuiltIn.Wait Until Keyword Succeeds 50x 3s Check Count Integrity ${switch_name} expected_controllers=3
+ ${new_owner} ${new_followers}= ClusterKeywords.Get Device Entity Owner And Followers Indexes ${active_session} openflow openflow:${idx}
+ BuiltIn.Should Be Equal ${old_owner} ${new_owner}
+ Collections.List Should Contain Value ${new_followers} ${old_follower}
+
+Rejoin Controller To The Cluster
+ [Arguments] ${isolated_node}
+ ClusterKeywords.Rejoin a Controller To Cluster ${isolated_node} @{CONTROLLER_NODES}
+ [Teardown] SSHLibrary.Switch Connection ${mininet_conn_id}
+
+Isolate Controller From The Cluster
+ [Arguments] ${isolated_node}
+ ClusterKeywords.Isolate a Controller From Cluster ${isolated_node} @{CONTROLLER_NODES}
+ [Teardown] SSHLibrary.Switch Connection ${mininet_conn_id}
+
+Check Count Integrity
+ [Arguments] ${switch_name} ${expected_controllers}=3
+ [Documentation] Every switch must have only one master and rest must be followers and together must be of expected nodes count
+ ${idx}= BuiltIn.Evaluate "${switch_name}"[1:]
+ ${owner} ${candidates}= ClusterKeywords.Get Device Entity Owner And Candidates Indexes ${active_session} openflow openflow:${idx}
+ ${count}= BuiltIn.Get Length ${candidates}
+ BuiltIn.Should Be Equal As Numbers ${expected_controllers} ${count}
+
+Verify New Master Controller Node
+ [Arguments] ${switch_name} ${old_master}
+ [Documentation] Checks if given node is different from actual master
+ ${idx}= BuiltIn.Evaluate "${switch_name}"[1:]
+ ${owner} ${followers}= ClusterKeywords.Get Device Entity Owner And Followers Indexes ${active_session} openflow openflow:${idx}
+ ${new_master} BuiltIn.Set Variable ${ODL_SYSTEM_${owner}_IP}
+ BuiltIn.Should Not Be Equal ${old_master} ${new_master}
+ Return From Keyword ${new_master}
+
+Verify Follower Added
+ [Arguments] ${switch_name} ${expected_node}
+ [Documentation] Checks if given node is in the list of active followers
+ ${idx}= BuiltIn.Evaluate "${switch_name}"[1:]
+ ${owner} ${followers}= ClusterKeywords.Get Device Entity Owner And Followers Indexes ${active_session} openflow openflow:${idx}
+ Collections.List Should Contain Value ${followers} ${expected_node}
--- /dev/null
+*** Settings ***
+Documentation Test suite for entity ownership service and openflowplugin. Makes changes on controller side (restart karaf)
+Suite Setup Start Suite
+Suite Teardown End Suite
+Test Template Restarting Karaf Scenario
+Library SSHLibrary
+Library RequestsLibrary
+Library XML
+Resource ${CURDIR}/../../../libraries/Utils.robot
+Resource ${CURDIR}/../../../libraries/OvsManager.robot
+Resource ${CURDIR}/../../../libraries/ClusterKeywords.robot
+Library Collections
+
+*** Variables ***
+${SWITCHES} 1
+# this is for mininet 2.2.1 ${START_CMD} sudo mn --controller=remote,ip=${ODL_SYSTEM_1_IP} --controller=remote,ip=${ODL_SYSTEM_2_IP} --controller=remote,ip=${ODL_SYSTEM_3_IP} --topo linear,${SWITCHES} --switch ovsk,protocols=OpenFlow13
+${START_CMD} sudo mn --topo linear,${SWITCHES} --switch ovsk,protocols=OpenFlow13
+${START_TIMEOUT} 90s
+${KARAF_HOME} ${WORKSPACE}${/}${BUNDLEFOLDER}
+
+*** Test Cases ***
+Switches To Be Connected To All Nodes
+ [Documentation] Initial check for correct connected topology.
+ [Template] NONE
+ BuiltIn.Wait Until Keyword Succeeds 5x 3s Check All Switches Connected To All Cluster Nodes
+
+Restarting Owner Of Switch s1
+ s1
+
+Switches Still Be Connected To All Nodes
+ [Template] NONE
+ BuiltIn.Wait Until Keyword Succeeds 5x 3s Check All Switches Connected To All Cluster Nodes
+
+*** Keywords ***
+Start Suite
+ BuiltIn.Log Start the test on the base edition
+ ${mininet_conn_id}= SSHLibrary.Open Connection ${TOOLS_SYSTEM_IP} prompt=${TOOLS_SYSTEM_PROMPT}
+ BuiltIn.Set Suite Variable ${mininet_conn_id}
+ SSHLibrary.Login With Public Key ${TOOLS_SYSTEM_USER} ${USER_HOME}/.ssh/id_rsa any
+ SSHLibrary.Put File ${CURDIR}/../../../libraries/DynamicMininet.py .
+ SSHLibrary.Execute Command sudo ovs-vsctl set-manager ptcp:6644
+ SSHLibrary.Execute Command sudo mn -c
+ SSHLibrary.Write ${START_CMD}
+ SSHLibrary.Read Until mininet>
+ ${cntls_list} BuiltIn.Create List ${ODL_SYSTEM_1_IP} ${ODL_SYSTEM_2_IP} ${ODL_SYSTEM_3_IP}
+ ${switch_list} BuiltIn.Create List
+ : FOR ${i} IN RANGE 0 ${SWITCHES}
+ \ ${sid}= BuiltIn.Evaluate ${i}+1
+ \ Collections.Append To List ${switch_list} s${sid}
+ : FOR ${i} IN RANGE 0 ${NUM_ODL_SYSTEM}
+ \ ${cid}= BuiltIn.Evaluate ${i}+1
+ \ RequestsLibrary.Create Session controller${cid} http://${ODL_SYSTEM_${cid}_IP}:${RESTCONFPORT} auth=${AUTH} headers=${HEADERS_XML}
+ BuiltIn.Set Suite Variable ${active_session} controller1
+ OvsManager.Setup Clustered Controller For Switches ${switch_list} ${cntls_list}
+ BuiltIn.Wait Until Keyword Succeeds 10s 1s Are Switches Connected Topo
+
+End Suite
+ RequestsLibrary.Delete All Sessions
+ Utils.Stop Suite
+
+Are Switches Connected Topo
+ [Documentation] Checks wheather switches are connected to controller
+ ${resp}= RequestsLibrary.Get Request ${active_session} ${OPERATIONAL_TOPO_API}/topology/flow:1 headers=${ACCEPT_XML}
+ BuiltIn.Log ${resp.content}
+ ${count}= XML.Get Element Count ${resp.content} xpath=node
+ BuiltIn.Should Be Equal As Numbers ${count} ${SWITCHES}
+
+Check All Switches Connected To All Cluster Nodes
+ [Documentation] Verifies all switches are connected to all cluster nodes
+ OvsManager.Get Ovsdb Data
+ : FOR ${i} IN RANGE 0 ${SWITCHES}
+ \ ${sid}= BuiltIn.Evaluate ${i}+1
+ \ OvsManager.Should Be Connected s${sid} ${ODL_SYSTEM_1_IP} update_data=${False}
+ \ OvsManager.Should Be Connected s${sid} ${ODL_SYSTEM_2_IP} update_data=${False}
+ \ OvsManager.Should Be Connected s${sid} ${ODL_SYSTEM_3_IP} update_data=${False}
+
+Restarting Karaf Scenario
+ [Arguments] ${switch_name}
+ [Documentation] Disconnect and connect master and follower and check switch data to be consistent
+ ${idx}= BuiltIn.Evaluate str("${switch_name}"[1:])
+ BuiltIn.Set Test Variable ${idx}
+ Kill Switchs Old Owner ${switch_name}
+ Restart Switchs Old Owner ${switch_name}
+ Kill Switchs Candidate ${switch_name}
+ Restart Switchs Candidate ${switch_name}
+ [Teardown] Run Keyword If "${stopped_karaf}"!="${Empty}" Start Controller Node And Verify ${stopped_karaf}
+
+Kill Switchs Old Owner
+ [Arguments] ${switch_name}
+ BuiltIn.Set Test Variable ${stopped_karaf} ${Empty}
+ Check Count Integrity ${switch_name} expected_controllers=3
+ ${old_owner} ${old_followers}= ClusterKeywords.Get Device Entity Owner And Followers Indexes ${active_session} openflow openflow:${idx}
+ ${old_master}= BuiltIn.Set Variable ${ODL_SYSTEM_${old_owner}_IP}
+ ${tmp_follower}= Collections.Get From List ${old_followers} 0
+ BuiltIn.Set Suite Variable ${active_session} controller${tmp_follower}
+ Stop Controller Node And Verify ${old_master}
+ BuiltIn.Set Test Variable ${stopped_karaf} ${old_master}
+ ${new_master}= BuiltIn.Wait Until Keyword Succeeds 5x 3s Verify New Master Controller Node ${switch_name} ${old_master}
+ ${owner} ${followers}= ClusterKeywords.Get Device Entity Owner And Followers Indexes ${active_session} openflow openflow:${idx}
+ Collections.List Should Contain Value ${old_followers} ${owner}
+ Check Count Integrity ${switch_name} expected_controllers=2
+ BuiltIn.Should Be Equal As Strings ${new_master} ${ODL_SYSTEM_${owner}_IP}
+ BuiltIn.Set Suite Variable ${active_session} controller${owner}
+ BuiltIn.Set Test Variable ${old_owner}
+ BuiltIn.Set Test Variable ${old_followers}
+ BuiltIn.Set Test Variable ${old_master}
+ BuiltIn.Set Test Variable ${owner}
+ BuiltIn.Set Test Variable ${new_master}
+
+Restart Switchs Old Owner
+ [Arguments] ${switch_name}
+ Start Controller Node And Verify ${old_master}
+ BuiltIn.Set Test Variable ${stopped_karaf} ${Empty}
+ BuiltIn.Wait Until Keyword Succeeds 5x 3s Verify Follower Added ${switch_name} ${old_owner}
+ ${new_owner} ${new_followers}= ClusterKeywords.Get Device Entity Owner And Followers Indexes ${active_session} openflow openflow:${idx}
+ Check Count Integrity ${switch_name} expected_controllers=3
+ BuiltIn.Should Be Equal ${owner} ${new_owner}
+ Collections.List Should Contain Value ${new_followers} ${old_owner}
+
+Kill Switchs Candidate
+ [Arguments] ${switch_name}
+ ${old_owner} ${old_followers}= ClusterKeywords.Get Device Entity Owner And Followers Indexes ${active_session} openflow openflow:${idx}
+ ${old_follower}= Collections.Get From List ${old_followers} 0
+ ${old_slave}= BuiltIn.Set Variable ${ODL_SYSTEM_${old_follower}_IP}
+ Stop Controller Node And Verify ${old_slave}
+ BuiltIn.Set Test Variable ${stopped_karaf} ${old_slave}
+ BuiltIn.Wait Until Keyword Succeeds 5x 3s Check Count Integrity ${switch_name} expected_controllers=2
+ ${owner} ${followers}= ClusterKeywords.Get Device Entity Owner And Followers Indexes ${active_session} openflow openflow:${idx}
+ BuiltIn.Should Be Equal ${owner} ${old_owner}
+ Collections.List Should Not Contain Value ${followers} ${old_follower}
+ BuiltIn.Should Be Equal As Strings ${new_master} ${ODL_SYSTEM_${owner}_IP}
+ BuiltIn.Set Test Variable ${old_owner}
+ BuiltIn.Set Test Variable ${old_followers}
+ BuiltIn.Set Test Variable ${old_follower}
+ BuiltIn.Set Test Variable ${old_slave}
+ BuiltIn.Set Test Variable ${owner}
+
+Restart Switchs Candidate
+ [Arguments] ${switch_name}
+ Start Controller Node And Verify ${old_slave}
+ BuiltIn.Set Test Variable ${stopped_karaf} ${Empty}
+ BuiltIn.Wait Until Keyword Succeeds 5x 3s Check Count Integrity ${switch_name} expected_controllers=3
+ ${new_owner} ${new_followers}= ClusterKeywords.Get Device Entity Owner And Followers Indexes ${active_session} openflow openflow:${idx}
+ BuiltIn.Should Be Equal ${old_owner} ${new_owner}
+ Collections.List Should Contain Value ${new_followers} ${old_follower}
+
+Check Count Integrity
+ [Arguments] ${switch_name} ${expected_controllers}=3
+ [Documentation] Every switch must have only one master and rest must be followers and together must be of expected nodes count
+ ${idx}= BuiltIn.Evaluate "${switch_name}"[1:]
+ ${owner} ${candidates}= ClusterKeywords.Get Device Entity Owner And Candidates Indexes ${active_session} openflow openflow:${idx}
+ ${count}= BuiltIn.Get Length ${candidates}
+ BuiltIn.Should Be Equal As Numbers ${expected_controllers} ${count}
+
+Verify New Master Controller Node
+ [Arguments] ${switch_name} ${old_master}
+ [Documentation] Checks if given node is different from actual master
+ ${idx}= BuiltIn.Evaluate "${switch_name}"[1:]
+ ${owner} ${followers}= ClusterKeywords.Get Device Entity Owner And Followers Indexes ${active_session} openflow openflow:${idx}
+ ${new_master} BuiltIn.Set Variable ${ODL_SYSTEM_${owner}_IP}
+ BuiltIn.Should Not Be Equal ${old_master} ${new_master}
+ Return From Keyword ${new_master}
+
+Verify Follower Added
+ [Arguments] ${switch_name} ${expected_node}
+ [Documentation] Checks if given node is in the list of active followers
+ ${idx}= BuiltIn.Evaluate "${switch_name}"[1:]
+ ${owner} ${followers}= ClusterKeywords.Get Device Entity Owner And Followers Indexes ${active_session} openflow openflow:${idx}
+ Collections.List Should Contain Value ${followers} ${expected_node}
+
+Stop Controller Node And Verify
+ [Arguments] ${node}
+ [Documentation] Stops the given node
+ ClusterKeywords.Stop Controller Node And Verify ${node}
+ [Teardown] SSHLibrary.Switch Connection ${mininet_conn_id}
+
+Start Controller Node And Verify
+ [Arguments] ${node}
+ [Documentation] Starts the given node
+ ClusterKeywords.Start Controller Node And Verify ${node} ${START_TIMEOUT}
+ [Teardown] SSHLibrary.Switch Connection ${mininet_conn_id}
integration/test/csit/suites/openflowplugin/Clustering
integration/test/csit/suites/openflowplugin/Clustering
integration/test/csit/suites/openflowplugin/Sanity3Node
+integration/test/csit/suites/openflowplugin/EntityOwhership/010_Switch_Disconnect.robot
+integration/test/csit/suites/openflowplugin/EntityOwhership/020_Cluster_Sync_Problems.robot
+integration/test/csit/suites/openflowplugin/EntityOwhership/030_Cluster_Node_Failure.robot
+
# Place the suites in run order:
integration/test/csit/suites/openflowplugin/Clustering
integration/test/csit/suites/openflowplugin/Clustering
+integration/test/csit/suites/openflowplugin/EntityOwhership/010_Switch_Disconnect.robot
+integration/test/csit/suites/openflowplugin/EntityOwhership/020_Cluster_Sync_Problems.robot
+integration/test/csit/suites/openflowplugin/EntityOwhership/030_Cluster_Node_Failure.robot
#Commenting this last test so we can better observe mininet stop bug in karaf console
#integration/test/csit/suites/openflowplugin/Sanity3Node
Code Review / integration / test.git / commitdiff