2 Documentation PCEP performance suite, uses restconf with configurable authentication.
4 ... Copyright (c) 2015 Cisco Systems, Inc. and others. All rights reserved.
6 ... This program and the accompanying materials are made available under the
7 ... terms of the Eclipse Public License v1.0 which accompanies this distribution,
8 ... and is available at http://www.eclipse.org/legal/epl-v10.html
13 ... This is a suite which has both scale and performance aspects.
14 ... Given scale target, suite reports failures if functional error
15 ... is detected, or if various time limits expire.
16 ... For passing test cases, their duration is the performance metric.
18 ... ODL acts as a translation layer between PCEP capable devices
19 ... and users employing RESTCONF.
20 ... Performance measurement focuses on two different workflows.
22 ... The first workflow is initial synchronization, when ODL learns
23 ... the state of PCEP topology as devices connect to it,
24 ... while restconf user reads the state repeatedly.
25 ... The second workflow is mass update, when restconf users issue RPCs
26 ... to updale Layer Switched Paths on Path Computation Clients.
28 ... This suite uses pcc-mock (downloaded from Nexus) to simulate PCCs.
29 ... It needs segment of bindable IP addresses,
30 ... one for each simulated PCC; so running pcc-mock from remote machine
31 ... is only viable when just single PCC is simulated.
32 ... Testing with multiple PCCs works best when pcc-mock
33 ... runs on the same VM as ODL, so 127.0.0.0/8 subnet can be used.
35 ... Library AuthStandalone is used directly for restconf reads
36 ... in the first workflow. That library transparently handles several
37 ... http authentication methods, based on credentials and pybot arguments.
39 ... In the second workflow, updater.py utility is used for issuing
40 ... rapid restconf requests. It can use multiple worker threads,
41 ... as http requests are blocking.
42 ... Due to CPython interpreter itself being single threaded,
43 ... amounts of threads above 8-16 are actually slightly slower
44 ... (which may roughly correspond to network traffic
45 ... being more limiting factor than CPU).
46 ... This suite starts updater utility bound to single CPU,
47 ... as this setup was the most performant in other tests.
49 ... In case of failed test case, other tests are skipped (unless
50 ... this is overriden by [Setup]) to finish test run sooner.
52 ... Variables and test case names refer to Controller and Mininet,
53 ... those are assumed to be separate remote VMs, one to host ODL,
54 ... other to host tools.
55 ... In case updater and pcc-mock are desired to run
56 ... from separate machines, their parameters use Mininet
57 ... values as default.
58 ... If both updater VM and pcc-mock VM parameters are specified,
59 ... Mininet parameters may be skipped.
61 ... Some launch scripts put restrictions on how pybot options
62 ... can be specified, so there are utility variables to help with
63 ... copying Controller related value to apply fo updater of pccmock.
64 ... Having a tool co-located with ODL reduces network latency,
65 ... but puts more pressure on CPU and memory on Controller VM.
67 ... In some environments, issues with TIME-WAIT prevent high restconf rates,
68 ... so TCP reuse is temporarily allowed during the suite run, if possible
69 ... (and if not disabled by UPDATERVM_ENABLE_TCP_RW_REUSE option value).
70 ... See http://vincent.bernat.im/en/blog/2014-tcp-time-wait-state-linux.html
71 ... This suite ignores possible failures when changing reuse.
73 ... Similarly, in some environments, handling of requests.Session object matters
74 ... try changing RESTCONF_REUSE value to see if it helps.
76 ... Variables to override (only if needed) in pybot command:
77 ... (Look into Variables table to see the default values.)
79 ... CONTROLLER: Numeric IP address of VM where ODL runs.
80 ... CONTROLLER_USER: Username for ssh login to ODL VM.
81 ... CONTROLLER_PASSWORD: Ssh password, empty means public keys are used instead.
82 ... CONTROLLER_PROMPT: Substring to identify Linux prompt on ODL VM.
83 ... CONTROLLER_WORKSPACE: Path to where files can be written on ODL VM.
84 ... FIRST_PCC_IP: Set in case bind address is different from public pcc-mock VM address.
85 ... LOG_NAME: Filename (without path) to save pcc-mock output into.
86 ... LOG_PATH: Override if not the same as pccmock VM workspace.
87 ... LSPS: Number of LSPs per PCC to simulate and test.
88 ... MININET: Numeric IP address of VM to run pcc-mock and updater from by default.
89 ... MININET_PASSWORD: Linux password to go with the username (empty means keys).
90 ... MININET_PROMPT: Substring to identify Linux prompt on Mininet VM.
91 ... MININET_USER: Linux username to SSH to on Mininet VM.
92 ... MININET_WORKSPACE: Path to where files may be created on Mininet VM.
93 ... MOCK_FILE: Filename to use for mock-pcc executable instead of the timestamped one.
94 ... PCCDOWNLOAD_HOSTHEADER: Download server may check checks this header before showing content.
95 ... PCCDOWNLOAD_URLBASE: URL to pcep-pcc-mock folder in Nexus (use numberic IP if DNS has problems).
96 ... PCCMOCK_COLOCATED: If True, set PCCMOCKVM* to mirror CONTROLLER*
97 ... PCCMOCKVM_IP: Override MININET for pcc-mock usage.
98 ... PCCMOCKVM_*: Override corresponding MININET_* for pcc-mock usage.
99 ... PCCS: Number of PCCs to simulate and test.
100 ... PCEP_READY_VERIFY_TIMEOUT: Grace period for pcep-topology to appear. Lower if ODL is ready.
101 ... RESTCONF_*: USER, PASSWORD and SCOPE to authenticate with, REUSE session.
102 ... (Note: If SCOPE is not empty, token-based authentication is used.)
103 ... UPDATER_COLOCATED: If True, overrides UPDATERVM_* parameters to point at Controller
104 ... (The purpose is to provide an option without ability to unpack CONTROLLER value.)
105 ... UPDATER_ODLADDRESS: Override if public CONTROLLER address is not best fit.
106 ... UPDATER_REFRESH: Main updater thread may sleep this long. Balance precision with overhead.
107 ... UPDATER_TIMEOUT: If updater stops itself if running more than this time.
108 ... (Set this limit according to your performance target.)
109 ... UPDATERVM_ENABLE_TCP_RW_REUSE: Set to false if changing Linux configuration is not desired.
110 ... UPDATERVM_IP: Override MININET for updater.py usage.
111 ... UPDATERVM_*: Override corresponding MININET_* for updater.py usage.
112 Suite Setup FailFast.Do_Not_Fail_Fast_From_Now_On
113 Suite Teardown Disconnect
114 Test Setup FailFast.Fail_This_Fast_On_Previous_Error
115 Test Teardown FailFast.Start_Failing_Fast_If_This_Failed
116 Variables ${CURDIR}/../../../variables/Variables.py
117 Library SSHLibrary timeout=10s
118 Library RequestsLibrary
119 Library ${CURDIR}/../../../libraries/AuthStandalone.py
120 Resource ${CURDIR}/../../../libraries/FailFast.robot
121 Resource ${CURDIR}/../../../libraries/NexusKeywords.robot # for Deploy_Artifact
122 Resource ${CURDIR}/../../../libraries/SSHKeywords.robot # for Require_* and Assure_*
123 Resource ${CURDIR}/../../../libraries/Utils.robot # for Flexible_SSH_Login
126 # This table acts as an exhaustive list of variables users can modify on pybot invocation.
127 # It also contains commented-out lines for variables defined elswhere.
128 # Keep this list in alphabetical order.
129 # ${CONTROLLER} is inherited from Variables.py
130 # ${CONTROLLER_USER} is inherited from Variables.py
131 # ${CONTROLLER_PASSWORD} is inherited from Variables.py
132 ${CONTROLLER_PROMPT} ${DEFAULT_LINUX_PROMPT} # from Variables.py
133 ${CONTROLLER_WORKSPACE} /tmp
134 ${FIRST_PCC_IP} ${PCCMOCKVM_IP}
135 # ${LOG_FILE} is reserved for location of pybot-created log.html
136 ${LOG_NAME} throughpcep.log
137 ${LOG_PATH} ${PCCMOCKVM_WORKSPACE}
140 # ${MININET_PASSWORD} is inherited from Variables.py
141 ${MININET_PROMPT} ${DEFAULT_LINUX_PROMPT} # from Variables.py
142 ${MININET_USER} mininet
143 ${MININET_WORKSPACE} /tmp
144 ${PCCDOWNLOAD_HOSTHEADER} nexus.opendaylight.org
145 ${PCCDOWNLOAD_URLBASE} http://${PCCDOWNLOAD_HOSTHEADER}/content/repositories/opendaylight.snapshot/org/opendaylight/bgpcep/pcep-pcc-mock/
146 ${PCCMOCK_COLOCATED} False
147 ${PCCMOCKVM_IP} ${MININET}
148 ${PCCMOCKVM_PASSWORD} ${MININET_PASSWORD}
149 ${PCCMOCKVM_PROMPT} ${MININET_PROMPT}
150 ${PCCMOCKVM_USER} ${MININET_USER}
151 ${PCCMOCKVM_WORKSPACE} ${MININET_WORKSPACE}
153 ${PCEP_READY_VERIFY_TIMEOUT} 300s
154 # Yes, the default timeout is 5 minutes, as this suite might be started eagerly just after ODL starts booting up.
155 ${RESTCONF_PASSWORD} ${PWD} # from Variables.py
156 ${RESTCONF_REUSE} True
157 ${RESTCONF_SCOPE} ${EMPTY}
158 ${RESTCONF_USER} ${USER} # from Variables.py
159 ${UPDATER_COLOCATED} False
160 ${UPDATER_ODLADDRESS} ${CONTROLLER}
161 ${UPDATER_REFRESH} 0.1
162 ${UPDATER_TIMEOUT} 300
163 ${UPDATERVM_ENABLE_TCP_RW_REUSE} True
164 ${UPDATERVM_IP} ${MININET}
165 ${UPDATERVM_PASSWORD} ${MININET_PASSWORD}
166 ${UPDATERVM_PROMPT} ${MININET_PROMPT}
167 ${UPDATERVM_USER} ${MININET_USER}
168 ${UPDATERVM_WORKSPACE} ${MININET_WORKSPACE}
172 [Documentation] SSH login to pcc-mock VM, download latest pcc-mock executable from Nexus.
173 BuiltIn.Run_Keyword_If ${PCCMOCK_COLOCATED} Pccmock_From_Controller
174 NexusKeywords.Initialize_Artifact_Deployment_And_Usage tools_system_connect=False
175 SSHLibrary.Open_Connection ${PCCMOCKVM_IP} alias=pccmock
176 SSHLibrary.Set_Client_Configuration timeout=10s
177 SSHLibrary.Set_Client_Configuration prompt=${PCCMOCKVM_PROMPT}
178 Utils.Flexible_SSH_Login ${PCCMOCKVM_USER} ${PCCMOCKVM_PASSWORD} delay=4s
179 ${file_name} = NexusKeywords.Deploy_Test_Tool bgpcep pcep-pcc-mock
180 BuiltIn.Set_Suite_Variable ${mock_location} ${file_name}
183 [Documentation] Open SSH session to updater VM, copy the utility there, including dependencies, also prepare direct http session.
184 BuiltIn.Run_Keyword_If ${UPDATER_COLOCATED} Updater_From_Controller
185 SSHLibrary.Open_Connection ${UPDATERVM_IP} alias=updater
186 SSHLibrary.Set_Client_Configuration timeout=20s
187 SSHLibrary.Set_Client_Configuration prompt=${UPDATERVM_PROMPT}
188 Utils.Flexible_SSH_Login ${UPDATERVM_USER} ${UPDATERVM_PASSWORD} delay=4s
189 SSHKeywords.Require_Python
190 SSHLibrary.Put_File ${CURDIR}/../../../../tools/pcep_updater/updater.py ${UPDATERVM_WORKSPACE}/
191 SSHLibrary.Put_File ${CURDIR}/../../../libraries/AuthStandalone.py ${UPDATERVM_WORKSPACE}/
192 SSHKeywords.Assure_Library_Counter target_dir=${UPDATERVM_WORKSPACE}
193 SSHKeywords.Assure_Library_Ipaddr target_dir=${UPDATERVM_WORKSPACE}
194 # Done preparation of Updater VM, now use AuthStandalone to create session from robot VM too.
195 BuiltIn.Log_Many ${RESTCONF_USER} ${RESTCONF_PASSWORD} ${RESTCONF_SCOPE} ${CONTROLLER}
196 ${session} = AuthStandalone.Init_Session ${CONTROLLER} ${RESTCONF_USER} ${RESTCONF_PASSWORD} ${RESTCONF_SCOPE}
197 BuiltIn.Set_Suite_Variable ${rest_session} ${session}
198 # TODO: Define http timeouts.
200 Save_And_Enable_Tcp_Rw_Reuse
201 [Documentation] If requested, temporarily enable TCP port reuse on Updater VM to allow for high rate of TCP connections. Do not start failing fast.
202 BuiltIn.Pass_Execution_If not ${UPDATERVM_ENABLE_TCP_RW_REUSE} Manipulation of tcp_rw_reuse is not requested.
203 ${old_value} = SSHLibrary.Execute_Command cat /proc/sys/net/ipv4/tcp_tw_reuse
204 # The next line may be skipped if the previous line failed.
205 BuiltIn.Set_Suite_Variable ${tcp_rw_reuse} ${old_value}
206 ${out} ${rc} = SSHLibrary.Execute_Command sudo bash -c "echo 1 > /proc/sys/net/ipv4/tcp_tw_reuse" return_rc=True
207 BuiltIn.Should_Be_Equal ${rc} ${0}
208 # Lack of sudo access should not prevent the rest of suite from trying without TCP reuse.
209 [Teardown] Do_Not_Start_Failing_If_This_Failed
211 Topology_Precondition
212 [Documentation] Verify that within timeout, PCEP topology is present, with no PCC connected.
215 Builtin.Wait_Until_Keyword_Succeeds ${PCEP_READY_VERIFY_TIMEOUT} 1s Pcep_Off
216 # Yes, timeout is 5 minutes, as this suite might be started eagerly just after ODL starts booting up.
219 [Documentation] Launch pcc-mock on background so simulated PCCs start connecting to controller.
220 SSHLibrary.Switch_Connection pccmock
221 ${command} = NexusKeywords.Compose_Full_Java_Command -jar ${mock_location} --local-address ${FIRST_PCC_IP} --remote-address ${CONTROLLER} --pcc ${PCCS} --lsp ${LSPS} &> ${LOG_PATH}/${LOG_NAME}
222 BuiltIn.Log ${command}
223 SSHLibrary.Write ${command}
224 # The pccmock SSH session is left alive, but no data will be exchanged for a while.
225 # We need this connection to stay alive to send ctrl+c later.
226 # SSHLibrary.Start_Command would not do that for us.
228 Topology_Intercondition
229 [Documentation] Verify that within timeout, PCEP topology contains correct numbers of LSPs.
231 ${localsize} = Evaluate int(${PCCS})*int(${LSPS})
232 Builtin.Set_Suite_Variable ${size} ${localsize}
234 Builtin.Wait_Until_Keyword_Succeeds 120s 1s Pcep_On
235 # TODO: Make timeout value scale with ${size}?
238 [Documentation] Run updater tool to change hops, using 1 blocking http thread.
241 [Teardown] Do_Not_Start_Failing_If_This_Failed
244 [Documentation] Verify that within timeout, the correct number of new hops is in PCEP topology.
249 [Documentation] Run updater tool to change hops again, using 2 blocking http threads.
252 [Teardown] Do_Not_Start_Failing_If_This_Failed
255 [Documentation] Verify that within timeout, the correct number of new hops is in PCEP topology.
260 [Documentation] Run updater tool to change hops again, using 4 blocking http threads.
263 [Teardown] Do_Not_Start_Failing_If_This_Failed
266 [Documentation] Verify that within timeout, the correct number of new hops is in PCEP topology.
271 [Documentation] Run updater tool to change hops again, using 8 blocking http threads.
274 [Teardown] Do_Not_Start_Failing_If_This_Failed
277 [Documentation] Verify that within timeout, the correct number of new hops is in PCEP topology.
282 [Documentation] Run updater tool to change hops again, using 16 blocking http threads.
285 [Teardown] Do_Not_Start_Failing_If_This_Failed
288 [Documentation] Verify that within timeout, the correct number of new hops is in PCEP topology.
293 [Documentation] Run updater tool to change hops again, using 32 blocking http threads.
296 [Teardown] Do_Not_Start_Failing_If_This_Failed
299 [Documentation] Verify that within timeout, the correct number of new hops is in PCEP topology.
304 [Documentation] Run updater tool to change hops again, using 64 blocking http threads.
307 [Teardown] Do_Not_Start_Failing_If_This_Failed
310 [Documentation] Verify that within timeout, the correct number of new hops is in PCEP topology.
315 [Documentation] Run updater tool to change hops again, using 128 blocking http threads.
318 [Teardown] Do_Not_Start_Failing_If_This_Failed
321 [Documentation] Verify that within timeout, the correct number of new hops is in PCEP topology.
326 [Documentation] Run updater tool to change hops again, using 256 blocking http threads.
329 [Teardown] Do_Not_Start_Failing_If_This_Failed
332 [Documentation] Verify that within timeout, the correct number of new hops is in PCEP topology.
337 [Documentation] Run updater tool to change hops again, using 512 blocking http threads.
340 [Teardown] Do_Not_Start_Failing_If_This_Failed
343 [Documentation] Verify that within timeout, the correct number of new hops is in PCEP topology.
348 [Documentation] Send ctrl+c to pcc-mock, see prompt again within timeout.
349 [Setup] Run_Even_When_Failing_Fast
350 # TODO: This should be in a library similar to BGPSpeaker.robot
351 SSHLibrary.Switch_Connection pccmock
352 # FIXME: send_ctrl should be in some library.
353 ${command} = BuiltIn.Evaluate chr(int(3))
354 BuiltIn.Log ${command}
355 SSHLibrary.Write ${command}
356 ${response} = SSHLibrary.Read_Until_Prompt
357 BuiltIn.Log ${response}
360 [Documentation] Transfer pcc-mock output from pcc-mock VM to robot VM.
361 [Setup] Run_Even_When_Failing_Fast
362 SSHLibrary.Get_File ${LOG_PATH}/${LOG_NAME} ${LOG_NAME}
364 Topology_Postcondition
365 [Documentation] Verify that within timeout, PCEP topology contains no PCCs again.
367 [Setup] Run_Even_When_Failing_Fast
368 Builtin.Wait_Until_Keyword_Succeeds 30s 1s Pcep_Off_Again
371 [Documentation] If requested, restore the old value if enabling TCP reuse was successful on Updater VM.
372 [Setup] Run_Even_When_Failing_Fast
373 BuiltIn.Pass_Execution_If not ${UPDATERVM_ENABLE_TCP_RW_REUSE} Manipulation of tcp_rw_reuse is not requested.
374 SSHLibrary.Switch_Connection updater
375 BuiltIn.Variable_Should_Exist ${tcp_rw_reuse}
376 ${out} ${rc} = SSHLibrary.Execute_Command sudo bash -c "echo ${tcp_rw_reuse} > /proc/sys/net/ipv4/tcp_tw_reuse" return_rc=True
377 BuiltIn.Should_Be_Equal ${rc} ${0}
380 Pccmock_From_Controller
381 [Documentation] Copy Controller values to Pccmock VM variables.
382 BuiltIn.Set_Suite_Variable ${PCCMOCKVM_IP} ${CONTROLLER}
383 BuiltIn.Set_Suite_Variable ${PCCMOCKVM_PASSWORD} ${CONTROLLER_PASSWORD}
384 BuiltIn.Set_Suite_Variable ${PCCMOCKVM_PROMPT} ${CONTROLLER_PROMPT}
385 BuiltIn.Set_Suite_Variable ${PCCMOCKVM_WORKSPACE} ${CONTROLLER_WORKSPACE}
386 BuiltIn.Set_Suite_Variable ${LOG_PATH} ${CONTROLLER_WORKSPACE}
388 Updater_From_Controller
389 [Documentation] Copy Controller values to Uprater VM variables.
390 BuiltIn.Set_Suite_Variable ${UPDATERVM_IP} ${CONTROLLER}
391 BuiltIn.Set_Suite_Variable ${UPDATERVM_PASSWORD} ${CONTROLLER_PASSWORD}
392 BuiltIn.Set_Suite_Variable ${UPDATERVM_PROMPT} ${CONTROLLER_PROMPT}
393 BuiltIn.Set_Suite_Variable ${UPDATERVM_WORKSPACE} ${CONTROLLER_WORKSPACE}
396 [Documentation] Explicitly close all SSH connections.
397 SSHLibrary.Close_All_Connections
398 # TODO: Make AuthStandalone session object closable?
400 Get_Pcep_Topology_Data
401 [Documentation] Use session object to download PCEP topology JSON. Check status and return Response object.
402 ${resp} = AuthStandalone.Get_Using_Session ${rest_session} operational/network-topology:network-topology/topology/pcep-topology
403 # Not Logging content, as it may be huge.
404 BuiltIn.Should_Be_Equal ${resp.status_code} ${200}
407 Get_Pcep_Topology_Count
408 [Arguments] ${pattern}
409 [Documentation] Get topology data, return number of pattern matches.
410 ${resp} = Get_Pcep_Topology_Data
411 # BuiltIn.Log ${resp.text}
412 ${count} = BuiltIn.Evaluate len(re.findall('${pattern}', '''${resp.text}''')) modules=re
417 [Documentation] Get topology data, Log content and assert the exact JSON of empty topology.
418 ${resp} = Get_Pcep_Topology_Data
419 # Used before topology had chance to grow huge. Be aware when creating a longevity suite from this.
420 BuiltIn.Log ${resp.text}
421 BuiltIn.Should_Be_Equal ${resp.text} {"topology":[{"topology-id":"pcep-topology","topology-types":{"network-topology-pcep:topology-pcep":{}}}]}
424 [Documentation] Get topology count of current hop, assert the number of matches.
425 # Suite variables ${size} and ${hop} are set elsewhere.
426 ${resp} = Get_Pcep_Topology_Count ${hop}
427 BuiltIn.Should_Be_Equal ${resp} ${size}
430 [Documentation] Get topology count of final hop, assert there is none.
431 ... This is more log friendly than Pcep_Off keyword, as it does not Log possibly large content.
433 ${resp} = Get_Pcep_Topology_Count ${hop}
434 BuiltIn.Should_Be_Equal ${resp} ${0}
437 [Arguments] ${iteration}
438 [Documentation] Set pattern to match the currently expected hop.
439 ${i} = BuiltIn.Evaluate str(1 + int(${iteration}))
440 # Regular Expressions need a dot to be escaped to represent a dot.
441 BuiltIn.Set_Suite_Variable ${hop} ${i}\.${i}\.${i}\.${i}/32
445 [Arguments] ${iteration}
446 [Documentation] Compute number of workers, call updater.py, assert its response.
447 SSHLibrary.Switch_Connection pccmock
448 # In some systems, inactive SSH sessions get severed.
449 ${command} = BuiltIn.Set_Variable echo "still alive"
450 ${output} = SSHLibrary.Execute_Command bash -c '${command}'
451 # The previous line relies on a fact that Execute_Command spawns separate shels, so running pcc-mock is not affected.
452 ${workers} = Evaluate 2**int(${iteration} - 1)
453 # TODO: Provide ${workers} explicitly as an argument to avoid math?
454 BuiltIn.Log ${workers}
456 SSHLibrary.Switch_Connection updater
457 ${response} = SSHLibrary.Execute_Command bash -c "cd ${UPDATERVM_WORKSPACE}; taskset 0x00000001 python updater.py --workers '${workers}' --odladdress '${UPDATER_ODLADDRESS}' --user '${RESTCONF_USER}' --password '${RESTCONF_PASSWORD}' --scope '${RESTCONF_SCOPE}' --pccaddress '${FIRST_PCC_IP}' --pccs '${PCCS}' --lsps '${LSPS}' --hop '${hop}' --timeout '${UPDATER_TIMEOUT}' --refresh '${UPDATER_REFRESH}' --reuse '${RESTCONF_REUSE}' 2>&1"
458 BuiltIn.Log ${response}
459 ${expected} = BuiltIn.Set_Variable Counter({'pass': ${size}})
460 BuiltIn.Log ${expected}
461 BuiltIn.Should_Contain ${response} ${expected}
464 [Arguments] ${iteration}
465 [Documentation] Set hop and verify that within timeout, all LSPs in topology are updated.
467 Builtin.Wait_Until_Keyword_Succeeds 30s 1s Pcep_On