Work on this patch was started by Jozef Behran.
Additional changes by Vratko Polak:
+ Refactored to use WaitUtils.
+ ConfigViaRestconf now tolerates multiple calls to its Setup.
+ Testplan includes netconf readiness suite at the start.
+ Documentation cleanup.
+ Minor improvements to WaitUtils and ScalarClosuresTest
+ Subset of ODL_SYSTEM* and TOOLS_SYSTEM* variables added to Variables.py
The original commit message by Jozef Behran:
The first version of the test suite (called
singlepeer_prefixcount.robot) can tell if the tested system
can meet a target but it is not able to give "pure ingestion"
performance data for the test load, especially when the test
load is large. The measurement gives a somewhat inaccurate
measurement of the time it needs to ingest the data while
the topology is being accessed along with time of the final
check of the topology lumped in. On the other side this
provides a "real world workload benchmark" number which may
be useful for some users of the testsuite. An additional
advantage of this suite is that it subjects the BGP feature
of ODL to additional stress, allowing for exposure of more
bugs.
The second suite (singlepeer_changecount.robot) is able to
give performance numbers (times of the actions of filling and
emptying the topology) as well as checking whether the target
is met because it uses change counter to do the topology
up-to-date-ness detection which is much more lightweight test
than downloading the entire topology and counting the routes
in it. However as it does not stress ODL so much, it has
increased chances of missing some bugs.
Change-Id: Ia54aa426370632ac4ee14e70a21fa29fa0addce3
Signed-off-by: Jozef Behran <jbehran@cisco.com>
Signed-off-by: Vratko Polak <vrpolak@cisco.com>
... and is available at http://www.eclipse.org/legal/epl-v10.html
...
...
-... This library assumes that a SSH connections exists (and is switched to)
-... to a Linux machine (usualy MININET) where the Python BGP speaker should be running.
+... This library assumes that a SSH connection exists (and is switched to)
+... to a Linux machine (usualy TOOLS_SYSTEM) where the Python BGP speaker should be run.
... It also assumes that the current working directory on that connection is the
... directory where the speaker tool was deployed as there are no paths to neither
... the play.py nor the log files in the commands.
+...
... TODO: The Utils.robot library has a "Run Command On Remote System" if we didn't
... want to make the assumption that an SSH connection was already open.
... alternative TODO: Explain that it is not worth to perform separate SSH logins.
Resource ${CURDIR}/Utils.robot
*** Variables ***
-${player_output_log} play.py.out
+${BGPSpeaker__OUTPUT_LOG} play.py.out
*** Keywords ***
Start_BGP_Speaker
... so it can be dumped into the logs later, when stopping it. This also avoids polluting the
... output seen by "Read Until Prompt" with false propmpts so it won't stop prematurely
... leading to a spurious test failure, messy log content or other misbehavior.
- ${command}= BuiltIn.Set_Variable python play.py ${arguments} &> ${player_output_log}
+ ${command} = BuiltIn.Set_Variable python play.py ${arguments} &> ${BGPSpeaker__OUTPUT_LOG}
BuiltIn.Log ${command}
- ${output}= SSHLibrary.Write ${command}
+ ${output} = SSHLibrary.Write ${command}
Dump_BGP_Speaker_Logs
[Documentation] Send all output produced by the play.py utility to Robot logs.
... This needs to be called if your suite detects play.py crashing and bypasses
... Kill_BGP_Speaker in that case otherwise the output of play.py (which most
... likely contains clues about why it crashed) will be lost.
- ${output_log}= SSHLibrary.Execute_Command cat ${player_output_log}
+ ${output_log} = SSHLibrary.Execute_Command cat ${BGPSpeaker__OUTPUT_LOG}
BuiltIn.Log ${output_log}
Kill_BGP_Speaker
[Documentation] Interrupt play.py, fail if no prompt is seen within SSHLibrary timeout.
... Also dump the logs with the output the program produced.
Utils.Write_Bare_Ctrl_C
- ${status} ${message}= BuiltIn.Run_Keyword_And_Ignore_Error SSHLibrary.Read_Until_Prompt
+ ${status} ${message} = BuiltIn.Run_Keyword_And_Ignore_Error SSHLibrary.Read_Until_Prompt
Dump_BGP_Speaker_Logs
+ # TODO: When Propagate_Failure is moved to better Resource, use it instead of the following.
BuiltIn.Return_From_Keyword_If '${status}' == 'PASS'
BuiltIn.Log ${message}
BuiltIn.Fail The prompt was not seen within timeout period.
--- /dev/null
+*** Settings ***
+Documentation Robot keyword library (Resource) for common handling of data change counter.
+...
+... Copyright (c) 2015 Cisco Systems, Inc. and others. All rights reserved.
+...
+... This program and the accompanying materials are made available under the
+... terms of the Eclipse Public License v1.0 which accompanies this distribution,
+... and is available at http://www.eclipse.org/legal/epl-v10.html
+...
+...
+... This resource assumes that RequestsLibrary has open a connection named "operational"
+... which points to (an analogue of) http://${ODL_SYSTEM_IP}:${RESTCONFPORT}/${OPERATIONAL_API}
+Library RequestsLibrary
+Resource ${CURDIR}/ConfigViaRestconf.robot
+Resource ${CURDIR}/ScalarClosures.robot
+Resource ${CURDIR}/WaitUtils.robot
+
+*** Variables ***
+${CHANGE_COUNTER_TEMPLATE_FOLDER} ${CURDIR}/../variables/bgpuser
+
+*** Keywords ***
+CC_Setup
+ [Documentation] Initialize dependency libraries.
+ ConfigViaRestconf.Setup_Config_Via_Restconf
+ WaitUtils.WU_Setup # includes ScalarClosures.SC_Setup
+ ${counter} = ScalarClosures.Closure_From_Keyword_And_Arguments Get_Change_Count
+ BuiltIn.Set_Suite_Variable ${ChangeCounter__getter} ${counter}
+
+Get_Change_Count
+ [Documentation] GET data change request, assert status 200, return the value.
+ ${response} = RequestsLibrary.Get_Request operational data-change-counter:data-change-counter
+ BuiltIn.Should_Be_Equal ${response.status_code} ${200} Got status: ${response.status_code} and message: ${response.text}
+ # TODO: The following line can be insecure. Should we use regexp instead?
+ ${count} = BuiltIn.Evaluate ${response.text}["data-change-counter"]["count"]
+ [Return] ${count}
+
+Reconfigure_Topology_Name
+ [Arguments] ${topology_name}=example-linkstate-topology
+ [Documentation] Configure data change counter to count transactions affecting
+ ... ${topology_name} instead of previously configured topology name.
+ ${template_as_string}= BuiltIn.Set_Variable {'TOPOLOGY_NAME': '${topology_name}'}
+ ConfigViaRestconf.Put_Xml_Template_Folder_Config_Via_Restconf ${CHANGE_COUNTER_TEMPLATE_FOLDER}${/}change_counter ${template_as_string}
+
+Wait_For_Change_Count_To_Become_Stable
+ [Arguments] ${timeout}=60s ${period}=1s ${repetitions}=4 ${count_to_overcome}=-1
+ [Documentation] Each ${period} get count. After ${repetitions} of constant value above ${count_to_overcome} within ${timeout}, Return validator output. Fail early on getter error.
+ ${validator} = WaitUtils.Create_Limiting_Stability_Safe_Stateful_Validator_From_Value_To_Overcome maximum_invalid=${count_to_overcome}
+ ${result} = WaitUtils.Wait_For_Getter_Error_Or_Safe_Stateful_Validator_Consecutive_Success timeout=${timeout} period=${period} count=${repetitions} getter=${ChangeCounter__getter} safe_validator=${validator}
+ ... initial_state=${count_to_overcome}
+ [Return] ${result}
...
... Prerequisities:
... * netconf-connector feature installed on ODL.
-... * Setup_Config_Via_Restconf called from suite Setup once
-... (or before any other keyword from this library, but just once).
+... * Setup_Config_Via_Restconf called from suite Setup
+... (or before any other call to a keyword from this library) at least once.
Library OperatingSystem
Library RequestsLibrary
Library String
Setup_Config_Via_Restconf
[Documentation] Creates Requests session to be used by subsequent keywords.
... Also remembers worspace to use when needed and two temp files for JSON data.
+ # Check for multiple Setup calls.
+ ${variable_was_set}= BuiltIn.Get_Variable_Value ${cvr_actfile} NEVER
+ BuiltIn.Return_From_Keyword_If '''${variable_was_set}''' != '''NEVER'''
# Do not append slash at the end uf URL, Requests would add another, resulting in error.
RequestsLibrary.Create_Session cvr_session http://${CONTROLLER}:${RESTCONFPORT}${CONTROLLER_CONFIG_MOUNT} headers=${HEADERS_XML} auth=${AUTH}
- ${workspace_defined} BuiltIn.Run_Keyword_And_return_Status BuiltIn.Variable_Should_Exist ${WORKSPACE}
+ ${workspace_defined}= BuiltIn.Run_Keyword_And_return_Status BuiltIn.Variable_Should_Exist ${WORKSPACE}
BuiltIn.Run_Keyword_If ${workspace_defined} BuiltIn.Set_Suite_Variable ${cvr_workspace} ${WORKSPACE}
BuiltIn.Set_Suite_Variable ${cvr_actfile} ${cvr_workspace}${/}actual.json
BuiltIn.Set_Suite_Variable ${cvr_expfile} ${cvr_workspace}${/}expected.json
${response}= RequestsLibrary.Post_Request cvr_session ${uri_part} data=${xml_data}
BuiltIn.Log ${response.text}
BuiltIn.Should_Be_Empty ${response.text}
+ # TODO: status_code is integrer, so compare to ${204}. Also, there is a Improvement for 201 to be a better code.
BuiltIn.Should_Be_Equal_As_Strings ${response.status_code} 204
--- /dev/null
+*** Settings ***
+Documentation Robot keyword library (Resource) for common BGP actions concerned with counting prefixes.
+...
+... Copyright (c) 2015 Cisco Systems, Inc. and others. All rights reserved.
+...
+... This program and the accompanying materials are made available under the
+... terms of the Eclipse Public License v1.0 which accompanies this distribution,
+... and is available at http://www.eclipse.org/legal/epl-v10.html
+...
+...
+... Currently, all keywords count prefixes only in example-ipv4-topology.
+... Prefix is identified by simplistic regular expression on JSON data.
+...
+... This resource assumes that RequestsLibrary has open a connection named "operational"
+... which points to (an analogue of) http://${ODL_SYSTEM_IP}:${RESTCONFPORT}/${OPERATIONAL_API}
+Library RequestsLibrary
+Resource ${CURDIR}/WaitUtils.robot
+
+*** Keywords ***
+PC_Setup
+ [Documentation] Call dependency setups and construct suite variables.
+ WaitUtils.WU_Setup # includes ScalarClosures.SC_Setup
+ ${getter} = ScalarClosures.Closure_From_Keyword_And_Arguments Get_Ipv4_Topology_Count
+ BuiltIn.Set_Suite_Variable ${PrefixCounting__getter} ${getter}
+
+Get_Ipv4_Topology
+ [Documentation] GET the example-ipv4-topology data, check status is 200, return the topology data.
+ ...
+ ... Contrary to Utils.Get_Data_From_URI, this does not Log the (potentially huge) content.
+ ${response} = RequestsLibrary.Get_Request operational network-topology:network-topology/topology/example-ipv4-topology
+ Run_Keyword_If ${response.status_code} != 200 Fail Get on example-ipv4-topology returned status code ${response.status_code} with message: ${response.text}
+ [Return] ${response.text}
+
+Get_Ipv4_Topology_Count
+ [Documentation] Get topology. If not fail, return number of prefixes in the topology.
+ ${topology} = Get_Ipv4_Topology
+ # Triple quotes are precaution against formatted output.
+ ${prefix_count} = Builtin.Evaluate len(re.findall('"prefix":"', '''${topology}''')) modules=re
+ [Return] ${prefix_count}
+
+Check_Ipv4_Topology_Count
+ [Arguments] ${expected_count}
+ [Documentation] Check that the count of prefixes matches the expected count. Fails if it does not. In either case, collect garbage.
+ ${actual_count} = ScalarClosures.Run_Keyword_And_Collect_Garbage Get_Ipv4_Topology_Count
+ BuiltIn.Should_Be_Equal_As_Strings ${actual_count} ${expected_count}
+
+Check_Ipv4_Topology_Is_Empty
+ [Documentation] Example_Ipv4_Topology has to give status 200 with zero prefixes.
+ ...
+ ... Functional suites should use a more strict Keyword which tests for the whole JSON structure.
+ Check_Ipv4_Topology_Count 0
+
+Wait_For_Ipv4_Topology_Prefixes_To_Become_Stable
+ [Arguments] ${timeout}=60s ${period}=5s ${repetitions}=1 ${excluded_count}=-1
+ [Documentation] Each ${period} get prefix count. After ${repetitions} of stable different from ${excluded_count} within ${timeout}, Return validator output. Fail early on getter error.
+ # This is very similar to ChangeCounter keyword, but attempt to extract common code would increase overall code size.
+ ${validator} = ScalarClosures.Closure_From_Keyword_And_Arguments WaitUtils.Excluding_Stability_Safe_Stateful_Validator_As_Keyword state_holder data_holder excluded_value=${excluded_count}
+ ${result} = WaitUtils.Wait_For_Getter_Error_Or_Safe_Stateful_Validator_Consecutive_Success timeout=${timeout} period=${period} count=${repetitions} getter=${PrefixCounting__getter} safe_validator=${validator}
+ ... initial_state=${excluded_count}
+ [Return] ${result}
${validator} = ScalarClosures.Closure_From_Keyword_And_Arguments WaitUtils.Limiting_Stability_Safe_Stateful_Validator_As_Keyword state_holder data_holder valid_minimum=${valid_minimum}
[Return] ${validator}
+Excluding_Stability_Safe_Stateful_Validator_As_Keyword
+ [Arguments] ${old_state} ${data} ${excluded_value}=-1
+ [Documentation] Report failure if got the excluded value or if data value changed from last time. Useful to become validator.
+ ${new_state} = BuiltIn.Set_Variable ${data}
+ BuiltIn.Return_From_Keyword_If ${data} == ${excluded_value} ${new_state} FAIL Got the excluded value.
+ BuiltIn.Return_From_Keyword_If ${data} != ${old_state} ${new_state} FAIL Data value has changed.
+ [Return] ${new_state} PASS Validated stable: ${data}
+
WaitUtils__Check_Sanity_And_Compute_Derived_Times
[Arguments] ${timeout}=60s ${period}=1s ${count}=1
[Documentation] Common checks for argument values. Return times in seconds and deadline date implied by timeout time.
\ # Getter may fail, but this Keyword should return state, so we need RKAIE.
\ ${status} ${data} = BuiltIn.Run_Keyword_And_Ignore_Error ScalarClosures.Run_Keyword_And_Collect_Garbage ScalarClosures.Run_Closure_As_Is ${getter}
\ BuiltIn.Return_From_Keyword_If '''${status}''' != '''PASS''' ${state} ${status} Getter failed: ${data}
- \ # TODO: Do we want to check time here?
+ \ # Is there enough time left?
+ \ ${status} ${message} = BuiltIn.Run_Keyword_And_Ignore_Error WaitUtils__Is_Deadline_Reachable date_deadline=${date_deadline} period_in_seconds=${period_in_seconds}
+ \ ... sleeps_left=${sleeps_left} message=Last result: ${result}
+ \ BuiltIn.Return_From_Keyword_If '''${status}''' != '''PASS''' ${state} ${status} ${message}
\ ${state} ${status} ${result} = ScalarClosures.Run_Keyword_And_Collect_Garbage ScalarClosures.Run_Closure_After_Replacing_First_Two_Arguments ${safe_validator}
\ ... ${state} ${data}
\ # Validator may have reported failure.
--- /dev/null
+# Copyright (c) 2015 Cisco Systems, Inc. and others. All rights reserved.
+#
+# This program and the accompanying materials are made available under the
+# terms of the Eclipse Public License v1.0 which accompanies this distribution,
+# and is available at http://www.eclipse.org/legal/epl-v10.html
+
+# Place the suites in run order:
+integration/test/csit/scripts/controllermem_6g.sh
--- /dev/null
+*** Settings ***
+Documentation BGP performance of ingesting from 1 iBGP peer, data change counter used.
+...
+... Copyright (c) 2015 Cisco Systems, Inc. and others. All rights reserved.
+...
+... This program and the accompanying materials are made available under the
+... terms of the Eclipse Public License v1.0 which accompanies this distribution,
+... and is available at http://www.eclipse.org/legal/epl-v10.html
+...
+...
+... This suite uses play.py as single iBGP peer.
+... The suite only looks at example-ipv4-topology, so RIB is not examined.
+...
+... This suite requires odl-bgpcep-data-change-counter to be installed so
+... make sure it is added to "install-features" of any jobs that are going
+... to invoke it.
+...
+... The suite consists of two halves, differing on which side initiates BGP connection.
+... Data change counter is a lightweight way to detect "work is being done".
+... WaitUtils provide a nice Keyword to wait for stability, but it needs
+... initial value, that is why Store_Change_Count appears just before work-inducing action.
+... The time for Wait_For_Stable_* cases to finish is the main performance metric.
+... After waiting for stability is done, full check on number of prefixes present is performed.
+...
+... TODO: Currently, if a bug causes zero increase of data changes,
+... affected test cases will wait for max time. Reconsider.
+... If zero increase is allowed as stable, higher number of repetitions should be required.
+...
+... Additionally this test suite is not compatible with Helium and Hydrogen
+... releases as they don't include data change counter feature.
+... Use the other version of the suite (singlepeer_prefixcount.robot) to test them.
+Suite Setup Setup_Everything
+Suite Teardown Teardown_Everything
+Test Setup SetupUtils.Setup_Test_With_Logging_And_Fast_Failing
+Test Teardown FailFast.Start_Failing_Fast_If_This_Failed
+Library SSHLibrary timeout=10s
+Library RequestsLibrary
+Variables ${CURDIR}/../../../variables/Variables.py
+Resource ${CURDIR}/../../../libraries/BGPSpeaker.robot
+Resource ${CURDIR}/../../../libraries/ChangeCounter.robot
+Resource ${CURDIR}/../../../libraries/ConfigViaRestconf.robot
+Resource ${CURDIR}/../../../libraries/FailFast.robot
+Resource ${CURDIR}/../../../libraries/KillPythonTool.robot
+Resource ${CURDIR}/../../../libraries/PrefixCounting.robot
+Resource ${CURDIR}/../../../libraries/SetupUtils.robot
+
+*** Variables ***
+${BGP_VARIABLES_FOLDER} ${CURDIR}/../../../variables/bgpuser/
+${HOLDTIME} 180
+${HOLDTIME_CHANGE_COUNT} ${HOLDTIME}
+${COUNT} 1000000
+${COUNT_CHANGE_COUNT} ${COUNT}
+${CHECK_PERIOD} 1
+${CHECK_PERIOD_CHANGE_COUNT} ${CHECK_PERIOD}
+${REPETITIONS_CHANGE_COUNT} 1
+# TODO: Option names can be better.
+${last_change_count} -1
+
+*** Test Cases ***
+Check_For_Empty_Ipv4_Topology_Before_Talking
+ [Documentation] Wait for example-ipv4-topology to come up and empty. Give large timeout for case when BGP boots slower than restconf.
+ [Tags] critical
+ # TODO: Choose which tags to assign and make sure they are assigned correctly.
+ BuiltIn.Wait_Until_Keyword_Succeeds 120s 1s PrefixCounting.Check_Ipv4_Topology_Is_Empty
+
+Reconfigure_ODL_To_Accept_Connection
+ [Documentation] Configure BGP peer module with initiate-connection set to false.
+ ${template_as_string} = BuiltIn.Set_Variable {'IP': '${TOOLS_SYSTEM_IP}', 'HOLDTIME': '${HOLDTIME_CHANGE_COUNT}', 'PEER_PORT': '${BGP_TOOL_PORT}', 'INITIATE': 'false'}
+ ConfigViaRestconf.Put_Xml_Template_Folder_Config_Via_Restconf ${BGP_VARIABLES_FOLDER}${/}bgp_peer ${template_as_string}
+
+Wait_For_Data_Change_Counter_Ready
+ [Documentation] Data change counter might have been slower to start than ipv4 topology, wait for it.
+ BuiltIn.Wait_Until_Keyword_Succeeds 60s 1s ChangeCounter.Get_Change_Count
+
+Reconfigure_Data_Change_Counter
+ [Documentation] Configure data change counter to count transactions in example-ipv4-topology instead of example-linkstate-topology.
+ ChangeCounter.Reconfigure_Topology_Name example-ipv4-topology
+
+Start_Talking_BGP_speaker
+ [Documentation] Start Python speaker to connect to ODL.
+ Store_Change_Count
+ # Myport value is needed for checking whether connection at precise port was established.
+ BGPSpeaker.Start_BGP_speaker --amount ${COUNT_CHANGE_COUNT} --myip=${TOOLS_SYSTEM_IP} --myport=${BGP_TOOL_PORT} --peerip=${ODL_SYSTEM_IP} --peerport=${ODL_BGP_PORT}
+
+Wait_For_Stable_Talking_Ipv4_Topology
+ [Documentation] Wait until example-ipv4-topology becomes stable. This is done by checking the change counter.
+ ChangeCounter.Wait_For_Change_Count_To_Become_Stable timeout=${bgp_filling_timeout} period=${CHECK_PERIOD_CHANGE_COUNT} repetitions=${REPETITIONS_CHANGE_COUNT} count_to_overcome=${last_change_count}
+
+Check_Talking_Ipv4_Topology_Count
+ [Documentation] Count the routes in example-ipv4-topology and fail if the count is not correct.
+ [Tags] critical
+ PrefixCounting.Check_Ipv4_Topology_Count ${COUNT_CHANGE_COUNT}
+
+Kill_Talking_BGP_Speaker
+ [Documentation] Abort the Python speaker. Also, attempt to stop failing fast.
+ [Setup] SetupUtils.Setup_Test_With_Logging_And_Without_Fast_Failing
+ Store_Change_Count
+ BGPSpeaker.Kill_BGP_Speaker
+ FailFast.Do_Not_Fail_Fast_From_Now_On
+ # NOTE: It is still possible to remain failing fast, if both previous and this test have failed.
+ [Teardown] FailFast.Do_Not_Start_Failing_If_This_Failed
+
+Wait_For_Stable_Ipv4_Topology_After_Talking
+ [Documentation] Wait until example-ipv4-topology becomes stable again.
+ [Tags] critical
+ ChangeCounter.Wait_For_Change_Count_To_Become_Stable timeout=${bgp_emptying_timeout} period=${CHECK_PERIOD_CHANGE_COUNT} repetitions=${REPETITIONS_CHANGE_COUNT} count_to_overcome=${last_change_count}
+
+Check_For_Empty_Ipv4_Topology_After_Talking
+ [Documentation] Example-ipv4-topology should be empty now.
+ [Tags] critical
+ PrefixCounting.Check_Ipv4_Topology_Is_Empty
+
+Start_Listening_BGP_Speaker
+ [Documentation] Start Python speaker in listening mode.
+ BGPSpeaker.Start_BGP_speaker --amount ${COUNT_CHANGE_COUNT} --listen --myip=${TOOLS_SYSTEM_IP} --myport=${BGP_TOOL_PORT} --peerip=${ODL_SYSTEM_IP}
+
+Reconfigure_ODL_To_Initiate_Connection
+ [Documentation] Replace BGP peer config module, now with initiate-connection set to true.
+ Store_Change_Count
+ ${template_as_string} = BuiltIn.Set_Variable {'IP': '${TOOLS_SYSTEM_IP}', 'HOLDTIME': '${HOLDTIME_CHANGE_COUNT}', 'PEER_PORT': '${BGP_TOOL_PORT}', 'INITIATE': 'true'}
+ ConfigViaRestconf.Put_Xml_Template_Folder_Config_Via_Restconf ${BGP_VARIABLES_FOLDER}${/}bgp_peer ${template_as_string}
+
+Wait_For_Stable_Listening_Ipv4_Topology
+ [Documentation] Wait until example-ipv4-topology becomes stable.
+ ChangeCounter.Wait_For_Change_Count_To_Become_Stable timeout=${bgp_filling_timeout} period=${CHECK_PERIOD_CHANGE_COUNT} repetitions=${REPETITIONS_CHANGE_COUNT} count_to_overcome=${last_change_count}
+
+Check_Listening_Ipv4_Topology_Count
+ [Documentation] Count the routes in example-ipv4-topology and fail if the count is not correct.
+ [Tags] critical
+ PrefixCounting.Check_Ipv4_Topology_Count ${COUNT_CHANGE_COUNT}
+
+Kill_Listening_BGP_Speaker
+ [Documentation] Abort the Python speaker. Also, attempt to stop failing fast.
+ [Setup] SetupUtils.Setup_Test_With_Logging_And_Without_Fast_Failing
+ Store_Change_Count
+ BGPSpeaker.Kill_BGP_Speaker
+ FailFast.Do_Not_Fail_Fast_From_Now_On
+ # NOTE: It is still possible to remain failing fast, if both previous and this test have failed.
+ [Teardown] FailFast.Do_Not_Start_Failing_If_This_Failed
+
+Wait_For_Stable_Ipv4_Topology_After_Listening
+ [Documentation] Wait until example-ipv4-topology becomes stable again.
+ [Tags] critical
+ ChangeCounter.Wait_For_Change_Count_To_Become_Stable timeout=${bgp_emptying_timeout} period=${CHECK_PERIOD_CHANGE_COUNT} repetitions=${REPETITIONS_CHANGE_COUNT} count_to_overcome=${last_change_count}
+
+Check_For_Empty_Ipv4_Topology_After_Listening
+ [Documentation] Example-ipv4-topology should be empty now.
+ [Tags] critical
+ PrefixCounting.Check_Ipv4_Topology_Is_Empty
+
+Restore_Data_Change_Counter_Configuration
+ [Documentation] Configure data change counter back to count transactions affecting example-linkstate-topology.
+ [Setup] SetupUtils.Setup_Test_With_Logging_And_Without_Fast_Failing
+ ChangeCounter.Reconfigure_Topology_Name example-linkstate-topology
+
+Delete_Bgp_Peer_Configuration
+ [Documentation] Revert the BGP configuration to the original state: without any configured peers.
+ [Setup] SetupUtils.Setup_Test_With_Logging_And_Without_Fast_Failing
+ ConfigViaRestconf.Delete_Xml_Template_Folder_Config_Via_Restconf ${BGP_VARIABLES_FOLDER}${/}bgp_peer
+
+*** Keywords ***
+Setup_Everything
+ [Documentation] Setup imported resources, SSH-login to mininet machine,
+ ... create HTTP session, put Python tool to mininet machine.
+ SetupUtils.Setup_Utils_For_Setup_And_Teardown
+ ConfigViaRestconf.Setup_Config_Via_Restconf
+ ChangeCounter.CC_Setup
+ PrefixCounting.PC_Setup
+ SSHLibrary.Set_Default_Configuration prompt=${TOOLS_SYSTEM_PROMPT}
+ SSHLibrary.Open_Connection ${TOOLS_SYSTEM_IP}
+ Utils.Flexible_Mininet_Login
+ RequestsLibrary.Create_Session operational http://${ODL_SYSTEM_IP}:${RESTCONFPORT}${OPERATIONAL_API} auth=${AUTH}
+ # TODO: Do not include slash in ${OPERATIONAL_TOPO_API}, having it typed here is more readable.
+ # TODO: Alternatively, create variable in Variables which starts with http.
+ # Both TODOs would probably need to update every suite relying on current Variables.
+ SSHLibrary.Put_File ${CURDIR}/../../../../tools/fastbgp/play.py
+ # Calculate the timeout value based on how many routes are going to be pushed
+ ${count} = Builtin.Convert_To_Integer ${COUNT_CHANGE_COUNT}
+ # TODO: Replace 20 with some formula from period and repetitions.
+ Builtin.Set_Suite_Variable ${bgp_filling_timeout} ${count*3/10000+20}
+ Builtin.Set_Suite_Variable ${bgp_emptying_timeout} ${bgp_filling_timeout*3/4}
+
+Teardown_Everything
+ [Documentation] Make sure Python tool was killed and tear down imported Resources.
+ KillPythonTool.Search_And_Kill_Remote_Python 'play\.py'
+ ConfigViaRestconf.Teardown_Config_Via_Restconf
+ RequestsLibrary.Delete_All_Sessions
+ SSHLibrary.Close_All_Connections
+
+Store_Change_Count
+ [Documentation] Get the count of changes from BGP change counter. Ignore error or store the value.
+ ${status} ${count} = BuiltIn.Run_Keyword_And_Ignore_Error ChangeCounter.Get_Change_Count
+ BuiltIn.Run_Keyword_If '${status}' == 'PASS' BuiltIn.Set_Suite_Variable ${last_change_count} ${count}
--- /dev/null
+*** Settings ***
+Documentation BGP performance of ingesting from 1 iBGP peer, data change counter NOT used.
+...
+... Copyright (c) 2015 Cisco Systems, Inc. and others. All rights reserved.
+...
+... This program and the accompanying materials are made available under the
+... terms of the Eclipse Public License v1.0 which accompanies this distribution,
+... and is available at http://www.eclipse.org/legal/epl-v10.html
+...
+...
+... This suite uses play.py as single iBGP peer.
+... The suite only looks at example-ipv4-topology, so RIB is not examined.
+...
+... The suite consists of two halves, differing on which side initiates BGP connection.
+... State of "work is being done" is detected by increasing value of prefixes in topology.
+... The time for Wait_For_Stable_* cases to finish is the main performance metric.
+... After waiting for stability is done, full check on number of prefixes present is performed.
+...
+... TODO: Currently, if a bug causes prefix count to remain at zero,
+... affected test cases will wait for max time. Reconsider.
+... If zero is allowed as stable, higher period or repetitions would be required.
+...
+... The prefix counting is quite heavyweight and may induce large variation in time.
+... Try the other version of the suite (singlepeer_changecount.robot) to get better precision.
+Suite Setup Setup_Everything
+Suite Teardown Teardown_Everything
+Test Setup SetupUtils.Setup_Test_With_Logging_And_Fast_Failing
+Test Teardown FailFast.Start_Failing_Fast_If_This_Failed
+Library SSHLibrary timeout=10s
+Library RequestsLibrary
+Variables ${CURDIR}/../../../variables/Variables.py
+Resource ${CURDIR}/../../../libraries/BGPSpeaker.robot
+Resource ${CURDIR}/../../../libraries/ConfigViaRestconf.robot
+Resource ${CURDIR}/../../../libraries/FailFast.robot
+Resource ${CURDIR}/../../../libraries/KillPythonTool.robot
+Resource ${CURDIR}/../../../libraries/PrefixCounting.robot
+Resource ${CURDIR}/../../../libraries/SetupUtils.robot
+
+*** Variables ***
+${BGP_VARIABLES_FOLDER} ${CURDIR}/../../../variables/bgpuser/
+${HOLDTIME} 180
+${HOLDTIME_PREFIX_COUNT} ${HOLDTIME}
+${COUNT} 1000000
+${COUNT_PREFIX_COUNT} ${COUNT}
+${CHECK_PERIOD} 1
+${CHECK_PERIOD_PREFIX_COUNT} ${CHECK_PERIOD}
+${REPETITIONS_PREFIX_COUNT} 1
+# TODO: Option names can be better.
+${last_prefix_count} -1
+
+*** Test Cases ***
+Check_For_Empty_Ipv4_Topology_Before_Talking
+ [Documentation] Wait for example-ipv4-topology to come up and empty. Give large timeout for case when BGP boots slower than restconf.
+ [Tags] critical
+ # TODO: Choose which tags to assign and make sure they are assigned correctly.
+ BuiltIn.Wait_Until_Keyword_Succeeds 120s 1s PrefixCounting.Check_Ipv4_Topology_Is_Empty
+
+Reconfigure_ODL_To_Accept_Connection
+ [Documentation] Configure BGP peer module with initiate-connection set to false.
+ ${template_as_string} = BuiltIn.Set_Variable {'IP': '${TOOLS_SYSTEM_IP}', 'HOLDTIME': '${HOLDTIME_PREFIX_COUNT}', 'PEER_PORT': '${BGP_TOOL_PORT}', 'INITIATE': 'false'}
+ ConfigViaRestconf.Put_Xml_Template_Folder_Config_Via_Restconf ${BGP_VARIABLES_FOLDER}${/}bgp_peer ${template_as_string}
+
+Start_Talking_BGP_speaker
+ [Documentation] Start Python speaker to connect to ODL.
+ # Myport value is needed for checking whether connection at precise port was established.
+ BGPSpeaker.Start_BGP_speaker --amount ${COUNT_PREFIX_COUNT} --myip=${TOOLS_SYSTEM_IP} --myport=${BGP_TOOL_PORT} --peerip=${ODL_SYSTEM_IP} --peerport=${ODL_BGP_PORT}
+
+Wait_For_Stable_Talking_Ipv4_Topology
+ [Documentation] Wait until example-ipv4-topology becomes stable. This is done by checking stability of prefix count.
+ PrefixCounting.Wait_For_Ipv4_Topology_Prefixes_To_Become_Stable timeout=${bgp_filling_timeout} period=${CHECK_PERIOD_PREFIX_COUNT} repetitions=${REPETITIONS_PREFIX_COUNT} excluded_count=0
+
+Check_Talking_Ipv4_Topology_Count
+ [Documentation] Count the routes in example-ipv4-topology and fail if the count is not correct.
+ [Tags] critical
+ PrefixCounting.Check_Ipv4_Topology_Count ${COUNT_PREFIX_COUNT}
+
+Kill_Talking_BGP_Speaker
+ [Documentation] Abort the Python speaker. Also, attempt to stop failing fast.
+ [Setup] SetupUtils.Setup_Test_With_Logging_And_Without_Fast_Failing
+ BGPSpeaker.Kill_BGP_Speaker
+ FailFast.Do_Not_Fail_Fast_From_Now_On
+ # NOTE: It is still possible to remain failing fast, if both previous and this test have failed.
+ [Teardown] FailFast.Do_Not_Start_Failing_If_This_Failed
+
+Wait_For_Stable_Ipv4_Topology_After_Talking
+ [Documentation] Wait until example-ipv4-topology becomes stable again.
+ [Tags] critical
+ # TODO: Is is possible to have failed at Check_Talking_Ipv4_Topology_Count and still have initial period of constant count?
+ # FIXME: If yes, do count here to get the initial value and use it (if nonzero).
+ # TODO: If yes, decide whether access to the FailFast state should have keyword or just variable name.
+ PrefixCounting.Wait_For_Ipv4_Topology_Prefixes_To_Become_Stable timeout=${bgp_filling_timeout} period=${CHECK_PERIOD_PREFIX_COUNT} repetitions=${REPETITIONS_PREFIX_COUNT} excluded_count=${COUNT_PREFIX_COUNT}
+
+Check_For_Empty_Ipv4_Topology_After_Talking
+ [Documentation] Example-ipv4-topology should be empty now.
+ [Tags] critical
+ PrefixCounting.Check_Ipv4_Topology_Is_Empty
+
+Start_Listening_BGP_Speaker
+ [Documentation] Start Python speaker in listening mode.
+ BGPSpeaker.Start_BGP_speaker --amount ${COUNT_PREFIX_COUNT} --listen --myip=${TOOLS_SYSTEM_IP} --myport=${BGP_TOOL_PORT} --peerip=${ODL_SYSTEM_IP}
+
+Reconfigure_ODL_To_Initiate_Connection
+ [Documentation] Replace BGP peer config module, now with initiate-connection set to true.
+ ${template_as_string} = BuiltIn.Set_Variable {'IP': '${TOOLS_SYSTEM_IP}', 'HOLDTIME': '${HOLDTIME_PREFIX_COUNT}', 'PEER_PORT': '${BGP_TOOL_PORT}', 'INITIATE': 'true'}
+ ConfigViaRestconf.Put_Xml_Template_Folder_Config_Via_Restconf ${BGP_VARIABLES_FOLDER}${/}bgp_peer ${template_as_string}
+
+Wait_For_Stable_Listening_Ipv4_Topology
+ [Documentation] Wait until example-ipv4-topology becomes stable.
+ PrefixCounting.Wait_For_Ipv4_Topology_Prefixes_To_Become_Stable timeout=${bgp_filling_timeout} period=${CHECK_PERIOD_PREFIX_COUNT} repetitions=${REPETITIONS_PREFIX_COUNT} excluded_count=0
+
+Check_Listening_Ipv4_Topology_Count
+ [Documentation] Count the routes in example-ipv4-topology and fail if the count is not correct.
+ [Tags] critical
+ PrefixCounting.Check_Ipv4_Topology_Count ${COUNT_PREFIX_COUNT}
+
+Kill_Listening_BGP_Speaker
+ [Documentation] Abort the Python speaker. Also, attempt to stop failing fast.
+ [Setup] SetupUtils.Setup_Test_With_Logging_And_Without_Fast_Failing
+ BGPSpeaker.Kill_BGP_Speaker
+ FailFast.Do_Not_Fail_Fast_From_Now_On
+ # NOTE: It is still possible to remain failing fast, if both previous and this test have failed.
+ [Teardown] FailFast.Do_Not_Start_Failing_If_This_Failed
+
+Wait_For_Stable_Ipv4_Topology_After_Listening
+ [Documentation] Wait until example-ipv4-topology becomes stable again.
+ [Tags] critical
+ PrefixCounting.Wait_For_Ipv4_Topology_Prefixes_To_Become_Stable timeout=${bgp_filling_timeout} period=${CHECK_PERIOD_PREFIX_COUNT} repetitions=${REPETITIONS_PREFIX_COUNT} excluded_count=${COUNT_PREFIX_COUNT}
+
+Check_For_Empty_Ipv4_Topology_After_Listening
+ [Documentation] Example-ipv4-topology should be empty now.
+ [Tags] critical
+ PrefixCounting.Check_Ipv4_Topology_Is_Empty
+
+Delete_Bgp_Peer_Configuration
+ [Documentation] Revert the BGP configuration to the original state: without any configured peers.
+ [Setup] SetupUtils.Setup_Test_With_Logging_And_Without_Fast_Failing
+ ConfigViaRestconf.Delete_Xml_Template_Folder_Config_Via_Restconf ${BGP_VARIABLES_FOLDER}${/}bgp_peer
+
+*** Keywords ***
+Setup_Everything
+ [Documentation] Setup imported resources, SSH-login to mininet machine,
+ ... create HTTP session, put Python tool to mininet machine.
+ SetupUtils.Setup_Utils_For_Setup_And_Teardown
+ ConfigViaRestconf.Setup_Config_Via_Restconf
+ PrefixCounting.PC_Setup
+ SSHLibrary.Set_Default_Configuration prompt=${TOOLS_SYSTEM_PROMPT}
+ SSHLibrary.Open_Connection ${TOOLS_SYSTEM_IP}
+ Utils.Flexible_Mininet_Login
+ RequestsLibrary.Create_Session operational http://${ODL_SYSTEM_IP}:${RESTCONFPORT}${OPERATIONAL_API} auth=${AUTH}
+ # TODO: Do not include slash in ${OPERATIONAL_TOPO_API}, having it typed here is more readable.
+ # TODO: Alternatively, create variable in Variables which starts with http.
+ # Both TODOs would probably need to update every suite relying on current Variables.
+ SSHLibrary.Put_File ${CURDIR}/../../../../tools/fastbgp/play.py
+ # Calculate the timeout value based on how many routes are going to be pushed
+ ${count} = Builtin.Convert_To_Integer ${COUNT_PREFIX_COUNT}
+ # TODO: Replace 20 with some formula from period and repetitions.
+ Builtin.Set_Suite_Variable ${bgp_filling_timeout} ${count*3/10000+20}
+ Builtin.Set_Suite_Variable ${bgp_emptying_timeout} ${bgp_filling_timeout*3/4}
+
+Teardown_Everything
+ [Documentation] Make sure Python tool was killed and tear down imported Resources.
+ KillPythonTool.Search_And_Kill_Remote_Python 'play\.py'
+ ConfigViaRestconf.Teardown_Config_Via_Restconf
+ RequestsLibrary.Delete_All_Sessions
+ SSHLibrary.Close_All_Connections
One_Kwarg_Nested_In_Zero_Args_Execution_Test
[Documentation] Inner closure takes kwarg, outer closure runs it as is. Result shows execution chain went well and kwarg was processed.
${inner_closure} = ScalarClosures.Closure_From_Keyword_And_Arguments BuiltIn.Create_Dictionary foo=bar
+ # As stated in Documentation of ScalarClosures in Current limitation,
+ # BuiltIn.Create_Directory will get empty list in @{args}, which will trigger deprecation warning.
+ # TODO: Investigate whether it is a genuine bug in Robot.
${outer_closure} = ScalarClosures.Closure_From_Keyword_And_Arguments ScalarClosures.Run_Closure_As_Is ${inner_closure}
${actual} = ScalarClosures.Run_Closure_As_Is ${outer_closure}
${expected} = BuiltIn.Create_Dictionary foo=bar
--- /dev/null
+# Copyright (c) 2015 Cisco Systems, Inc. and others. All rights reserved.
+#
+# This program and the accompanying materials are made available under the
+# terms of the Eclipse Public License v1.0 which accompanies this distribution,
+# and is available at http://www.eclipse.org/legal/epl-v10.html
+
+# Place the suites in run order:
+integration/test/csit/suites/netconf/ready
+integration/test/csit/suites/bgpcep/bgpingest/singlepeer_changecount.robot
+integration/test/csit/suites/bgpcep/bgpingest/singlepeer_prefixcount.robot
"""
# Global variables
-CONTROLLER = '127.0.0.1'
+ODL_SYSTEM_IP = '127.0.0.1'
+CONTROLLER = ODL_SYSTEM_IP
PORT = '8080'
RESTPORT = '8282'
RESTCONFPORT = '8181'
TOPO_TREE_LEVEL = 2
TOPO_TREE_DEPTH = 3
TOPO_TREE_FANOUT = 2
+ODL_SYSTEM_IP_LIST = ['ODL_SYSTEM_1_IP', 'ODL_SYSTEM_2_IP', 'ODL_SYSTEM_3_IP']
CONTROLLERS = ['CONTROLLER', 'CONTROLLER1', 'CONTROLLER2']
-CONTROLLER_PASSWORD = '' # empty means use keys
-MININET_PASSWORD = '' # empty means use keys
+ODL_SYSTEM_PASSWORD = '' # empty means use keys
+CONTROLLER_PASSWORD = ODL_SYSTEM_PASSWORD
+TOOLS_SYSTEM_PASSWORD = '' # empty means use keys
+MININET_PASSWORD = TOOLS_SYSTEM_PASSWORD
KEYFILE_PASS = 'any'
SSH_KEY = 'id_rsa'
CONTROLLER_STOP_TIMEOUT = 120 # Max number of seconds test will wait for a controller to stop
# VM Environment Variables
DEFAULT_LINUX_PROMPT = '>'
+ODL_SYSTEM_PROMPT = DEFAULT_LINUX_PROMPT
+CONTROLLER_PROMPT = ODL_SYSTEM_PROMPT
+TOOLS_SYSTEM_PROMPT = DEFAULT_LINUX_PROMPT
+MININET_PROMPT = TOOLS_SYSTEM_PROMPT
# Netconf variables
ODL_NETCONF_PORT = '2830'
--- /dev/null
+config:modules/module/odl-data-change-counter-cfg:data-change-counter-impl/data-change-counter-singleton
--- /dev/null
+<module xmlns="urn:opendaylight:params:xml:ns:yang:controller:config">
+ <type xmlns:prefix="urn:opendaylight:params:xml:ns:yang:controller:bgpcep:data:change:counter">prefix:data-change-counter-impl</type>
+ <name>data-change-counter-singleton</name>
+ <data-provider xmlns="urn:opendaylight:params:xml:ns:yang:controller:bgpcep:data:change:counter">
+ <type xmlns:binding="urn:opendaylight:params:xml:ns:yang:controller:md:sal:binding">binding:binding-async-data-broker</type>
+ <name>pingpong-binding-data-broker</name>
+ </data-provider>
+ <topology-name xmlns="urn:opendaylight:params:xml:ns:yang:controller:bgpcep:data:change:counter">$TOPOLOGY_NAME</topology-name>
+</module>
Code Review / integration / test.git / commitdiff