6 This step by step guide aims to help projects with the task of creating a
7 System Test job that runs in Continuous Integration.
9 A System Test job will normally install a controller distribution in one or
10 more VMs and will run a functionality test using some test tool (e.g. mininet).
11 This job will run periodically, tipically once or twice a day.
13 All projects defining top-level features (essential functionality) and that have
14 decided to use the OpenDaylight CI for system test must create system test jobs.
16 System test jobs rely on Robot Framework, this is because Robot FW provides:
18 * Structure for test creation and execution (e.g. test suites, test cases that
20 * Easy test debug (real time logs, etc...).
21 * Test reports in Jenkins.
23 For those projects creating system test, Integration group will provide:
25 * Robot Framework support and assistance.
26 * Review of system test code. The code will be pushed to integration/test git
27 (csit/suites/$project/).
28 * JJB templates to install controller and execute a robot test to verify a
29 project functionality (releng/builder git, jjb/integration/).
31 Create basic system test
32 ------------------------
33 Download Integration/Test Repository::
35 git clone ssh://${USERNAME}@git.opendaylight.org:29418/integration/test.git
38 Follow the instructions in pulling-and-pushing-the-code_ to know more about
39 pulling and pushing code.
41 Create a folder for your project robot test::
43 mkdir test/csit/suites/$project
44 cd test/csit/suites/$project
46 Replace $project with your project name.
48 Move your robot suites (test folders) into the project folder:
50 If you do not have any robot test yet, copy integration basic folder suite into
51 your folder. You can later improve this suite or replace it by your own suites::
53 cp -R test/csit/suites/integration/basic basic
55 This suite will verify Restconf is operational.
59 A test plan is a text file indicating which robot test suites (including
60 integration repo path) will be executed to test a project functionality::
62 vim test/csit/testplans/$project-$functionality.txt
64 Replace $project with your project name and $functionality with the
65 functionality you want to test.
67 If you took the basic test from integration, the test plan file should look
70 # Place the suites in run order:
71 integration/test/csit/suites/$project/basic
73 Save the changes and exit editor.
75 Optional: Version specific test plan
76 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
77 Integration/Test is not part of the simultaneous release, so the same suites are
78 used for testing all supported ODL versions. There may be API changes between
79 different releases of ODL, which may require different logic in your Robot
80 tests. If the difference is small, it is recommended to act upon value of
81 ODL_STREAM variable (e.g. "beryllium", "boron", "carbon", etc).
83 If the difference is big, you may want to use different list of suites in
84 testplan. One way is to define separate jobs with different functionality names.
85 But the more convenient way is to define stream-specific testplan. For example::
87 vim test/csit/testplans/$project-$functionality-boron.txt
89 would contain a list of suites for testing Boron, while
90 $project-$functionality.txt would still contain the default list (used for
91 streams without stream specific testplans).
93 Optional: Create a script or config plan
94 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
95 Sometimes the environment prepared by scripts in releng/builder is not suitable
96 as is, and there are changes to be done before controller is installed (script
97 plan) or before it is started (config plan). You may create as many bash scripts
98 as you need in test/csit/scripts/ and then list them in the scriplans or
101 vim test/csit/scriptplans/$project-$functionality.txt
103 Save and push Test changes
104 ^^^^^^^^^^^^^^^^^^^^^^^^^^
105 Add the changes and push them in the integration/test repo::
111 Create system test job
112 ----------------------
113 Download RelEng Builder repository::
115 git clone ssh://${USERNAME}@git.opendaylight.org:29418/releng/builder
118 Follow the instructions in pulling-and-pushing-the-code_ to know more about
119 pulling and pushing code.
121 Create a new file and modify the values according to your project::
123 vim jjb/$project/$project-csit-$functionality.yaml
125 For a Managed project it should look like this::
129 name: openflowplugin-csit-flow-services
134 project: 'openflowplugin'
136 # The functionality under test
146 branch: 'stable/oxygen'
148 branch: 'stable/nitrogen'
150 branch: 'stable/carbon'
151 karaf-version: 'karaf3'
157 # Features to install
159 odl-openflowplugin-flow-services-rest,
160 odl-openflowplugin-app-table-miss-enforcer,
161 odl-openflowplugin-nxm-extensions
163 # Robot custom options
168 * name: give some name like $project-csit-$functionality.
169 * jobs: replace 1node by 3node if your test is develop for 3node cluster.
170 * project: set your your project name here (e.g. openflowplugin).
171 * functionality: set the functionality you want to test (e.g. flow-services).
172 Note this has also to match the robot test plan name you defined in the earlier
173 section `<Create a test plan_>`_ (e.g. openflowplugin-flow-services.txt)
174 * stream: list the project branches you are going to generate system test. Only
175 last branch if the project is new.
176 * install: this specifies controller installation, 'only' means only features in
177 install-features will be installed, 'all' means all compatible features will
178 be installed on top (multi-project features test).
179 * install-features: list of features you want to install in controller separated
181 * robot-options: robot option you want to pass to the test separated by space.
183 For Unmanaged project, we need 2 extra parameters:
185 * trigger-jobs: Unmanaged CSIT will run after succesful project merge, so just
186 fill with '{project}-merge-{stream}'.
187 * bundle-url: Unmanaged CSIT uses project local distribution, you can get the
188 local distribution URL from the Jenkins merge job itself (see example below).
190 So in this case it should look like this::
194 name: usc-csit-channel
201 # The functionality under test
202 functionality: 'channel'
208 trigger-jobs: '{project}-merge-{stream}'
209 # yamllint disable-line rule:line-length
210 bundle-url: 'https://jenkins.opendaylight.org/releng/view/usc/job/usc-merge-fluorine/lastBuild/org.opendaylight.usc$usc-karaf/artifact/org.opendaylight.usc/usc-karaf/1.6.0-SNAPSHOT/usc-karaf-1.6.0-SNAPSHOT.zip'
216 # Features to install
217 install-features: 'odl-restconf,odl-mdsal-apidocs,odl-usc-channel-ui'
219 # Robot custom options
222 Save the changes and exit editor.
224 Optional: Change default tools image
225 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
226 By default a system test spins a tools VM that can be used to run some test tool
227 like mininet, netconf tool, BGP simulator, etc. The default values are listed
228 below and you only need to specify them if you are changing something, for
229 example "tools_system_count: 0" will skip the tools VM if you do not need it.
230 For a list of available images see images-list_::
234 name: openflowplugin-csit-flow-services
239 project: 'openflowplugin'
241 # The functionality under test
251 branch: 'stable/oxygen'
253 branch: 'stable/nitrogen'
255 branch: 'stable/carbon'
256 karaf-version: 'karaf3'
263 tools_system_image: 'ZZCI - Ubuntu 16.04 - mininet-ovs-28 - 20180301-1041'
265 # Features to install
267 odl-openflowplugin-flow-services-rest,
268 odl-openflowplugin-app-table-miss-enforcer,
269 odl-openflowplugin-nxm-extensions
271 # Robot custom options
274 Optional: Plot a graph from your job
275 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
276 Scalability and peformance tests not only PASS/FAIL but most important they
277 provide a number or value we want to plot in a graph and track over different
280 For that you can add the plot configuration like in this example below::
284 name: openflowplugin-csit-cbench
289 project: 'openflowplugin'
291 # The functionality under test
292 functionality: 'cbench'
299 branch: 'stable/oxygen'
301 branch: 'stable/nitrogen'
303 branch: 'stable/carbon'
304 karaf-version: 'karaf3'
311 tools_system_image: 'ZZCI - Ubuntu 16.04 - mininet-ovs-28 - 20180301-1041'
313 # Features to install
314 install-features: 'odl-openflowplugin-flow-services-rest,odl-openflowplugin-drop-test'
316 # Robot custom options
317 robot-options: '-v duration_in_secs:60 -v throughput_threshold:20000 -v latency_threshold:5000'
320 01-plot-title: 'Throughput Mode'
321 01-plot-yaxis: 'flow_mods/sec'
322 01-plot-group: 'Cbench Performance'
323 01-plot-data-file: 'throughput.csv'
324 02-plot-title: 'Latency Mode'
325 02-plot-yaxis: 'flow_mods/sec'
326 02-plot-group: 'Cbench Performance'
327 02-plot-data-file: 'latency.csv'
331 * There are up to 10 plots per job and every plot can track different values,
332 for example max, min, average recorded in a csv file. In the example above you
333 can skip the 02-* lines if you do not use second plot.
334 * plot-title: title for your plot.
335 * plot-yaxis: your measurement (xaxis is build # so no need to fill).
336 * plot-group: just a label, use the same in case you have 2 plots.
337 * plot-data-file: this is the csv file generated by robot framework and contains
338 the values to plot. Examples can be found in openflow-performance_.
340 Optional: Add Patch Test Job to verify project patches
341 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
342 With the steps above your new csit job will run daily on latest generated
343 distribution. There is one more extra and optional step if you also want to run
344 your system test to verify patches in your project.
346 The patch test is triggered in gerrit using the keyword::
348 test-$project-$feature
352 * Build the gerrit patch.
353 * Create a distribution containing the patch.
354 * Trigger some system test (csit) that already exists and you specify with the
355 $feature definition below.
357 Create $project-patch-test.yaml file in your jjb folder::
359 vim jjb/$project/$project-patch-test-jobs.yaml
361 Fill the information as below::
365 name: openflowplugin-patch-test
370 project: 'openflowplugin'
378 branch: 'stable/oxygen'
381 branch: 'stable/nitrogen'
384 branch: 'stable/carbon'
386 karaf-version: 'karaf3'
393 openflowplugin-csit-1node-gate-flow-services-all-{stream},
394 openflowplugin-csit-1node-gate-scale-only-{stream},
395 openflowplugin-csit-1node-gate-perf-stats-collection-only-{stream},
396 openflowplugin-csit-1node-gate-perf-bulkomatic-only-{stream},
397 openflowplugin-csit-3node-gate-clustering-only-{stream},
398 openflowplugin-csit-3node-gate-clustering-bulkomatic-only-{stream},
399 openflowplugin-csit-3node-gate-clustering-perf-bulkomatic-only-{stream}
403 netvirt-csit-1node-openstack-{os-branch}-gate-stateful-{stream}
407 netvirt-csit-3node-openstack-{os-branch}-gate-stateful-{stream}
411 * name: give some name like $project-patch-test.
412 * project: set your your project name here (e.g. openflowplugin).
413 * stream: list the project branches you are going to generate system test. Only
414 last branch if the project is new.
415 * feature: you can group system tests in features. Note there is a predefined
416 feature -all- that triggers all features together.
417 * Fill the csit-list with all the system test jobs you want to run to verify a
422 Before pushing your system test job into jenkins-releng_, it is recommended to
423 debug the job as well as the you system test code in the sandbox. To do that:
425 * Set up sandbox access using jenkins-sandbox-install_ instruction.
426 * Push your new csit job to sandbox:
430 you can write a comment in a releng/builder gerrit patch to have the job automatically created
431 in the sandbox. The format of the comment is::
433 jjb-deploy <job name>
437 jenkins-jobs --conf jenkins.ini update jjb/ $project-csit-1node-$functionality-only-$branch
439 * Open your job in jenkins-sandbox_ and start a build replacing the PATCHREFSPEC
440 parameter by your int/test patch REFSPEC (e.g. refs/changes/85/23185/1). you
441 can find this info in gerrit top right corner 'Download' button.
442 * Update the PATCHREFSPEC parameter every time you push a new patchset in the
445 Optional: Debug VM issues in sandbox
446 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
447 In case of problems with the test VMs, you can easily debug these issues in the
448 sandbox by adding the following lines in a Jenkins shell window::
450 cat > ${WORKSPACE}/debug-script.sh <<EOF
452 <<put your debug shell script here>>
455 scp ${WORKSPACE}/debug-script.sh ${TOOLS_SYSTEM_IP}:/tmp
456 ssh ${TOOLS_SYSTEM_IP} 'sudo bash /tmp/debug-script.sh'
458 Note this will run a self-made debug script with sudo access in a VM of your
459 choice. In the example above you debug on the tools VM (TOOLS_SYSTEM_IP),
460 use ODL_SYSTEM_IP to debug in controller VM.
462 Save and push JJB changes
463 ^^^^^^^^^^^^^^^^^^^^^^^^^
464 Once you are happy with your system test, save the changes and push them in the
465 releng builder repo::
473 If this is your first system test job, it is recommended to add the int/test
474 patch (gerrit link) in the commit message so that committers can merge both
475 the int/test and the releng/builder patches at the same time.
477 Check system test jobs in Jenkins
478 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
479 Once your patches are merged your system test can be browsed in jenkins-releng_:
481 * $project-csit-1node-$functionality-only-$branch -> The single-feature test.
482 * $project-csit-1node-$functionality-all-$branch -> The multi-project test.
483 * $yourproject-patch-test-$feature-$branch -> Patch test job.
485 Note that jobs in jenkins-releng_ cannot be reconfigured, only jobs in
486 jenkins-sandbox_ can, that is why it is so important for testers to get access
491 Integration people are happy to support with questions and recommendations:
493 * Integration IRC: OpenDaylight channel 'opendaylight-integration
494 * Integration Mail: OpenDaylight list 'integration-dev@lists.opendaylight.org'
496 .. _pulling-and-pushing-the-code: http://docs.opendaylight.org/en/stable-boron/developer-guide/pulling-and-pushing-the-code-from-the-cli.html
497 .. _images-list: http://docs.opendaylight.org/en/stable-boron/submodules/releng/builder/docs/jenkins.html#pool-odlpub-hot-heat-orchestration-templates
498 .. _openflow-performance: https://git.opendaylight.org/gerrit/gitweb?p=integration/test.git;a=blob;f=csit/suites/openflowplugin/Performance/010_Cbench.robot
499 .. _jenkins-releng: https://jenkins.opendaylight.org/releng/
500 .. _jenkins-sandbox: https://jenkins.opendaylight.org/sandbox/
501 .. _jenkins-sandbox-install: http://docs.opendaylight.org/en/stable-boron/submodules/releng/builder/docs/jenkins.html#jenkins-sandbox