--- /dev/null
+System Test Guide
+=================
+
+Introduction
+------------
+This step by step guide aims to help projects with the task of creating a
+System Test job that runs in Continuous Integration.
+
+A System Test job will normally install a controller distribution in one or
+more VMs and will run a functionality test using some test tool (e.g. mininet).
+This job will run periodically, tipically once or twice a day.
+
+All projects defining top-level features (essential functionality) and that have
+decided to use the OpenDaylight CI for system test must create system test jobs.
+
+System test jobs rely on Robot Framework, this is because Robot FW provides:
+
+* Structure for test creation and execution (e.g. test suites, test cases that
+ PASS/FAIL).
+* Easy test debug (real time logs, etc...).
+* Test reports in Jenkins.
+
+For those projects creating system test, Integration group will provide:
+
+* Robot Framework support and assistance.
+* Review of system test code. The code will be pushed to integration/test git
+ (csit/suites/$project/).
+* JJB templates to install controller and execute a robot test to verify a
+ project functionality (releng/builder git, jjb/integration/).
+
+Create basic system test
+------------------------
+Download Integration/Test Repository::
+
+ git clone ssh://${USERNAME}@git.opendaylight.org:29418/integration/test.git
+ cd test
+
+Follow the instructions in pulling-and-pushing-the-code_ to know more about
+pulling and pushing code.
+
+Create a folder for your project robot test::
+
+ mkdir test/csit/suites/$project
+ cd test/csit/suites/$project
+
+Replace $project with your project name.
+
+Move your robot suites (test folders) into the project folder:
+
+If you do not have any robot test yet, copy integration basic folder suite into
+your folder. You can later improve this suite or replace it by your own suites::
+
+ cp -R test/csit/suites/integration/basic basic
+
+This suite will verify Restconf is operational.
+
+Create a test plan
+^^^^^^^^^^^^^^^^^^
+A test plan is a text file indicating which robot test suites (including
+integration repo path) will be executed to test a project functionality::
+
+ vim test/csit/testplans/$project-$functionality.txt
+
+Replace $project with your project name and $functionality with the
+functionality you want to test.
+
+If you took the basic test from integration, the test plan file should look
+like this::
+
+ # Place the suites in run order:
+ integration/test/csit/suites/$project/basic
+
+Save the changes and exit editor.
+
+Optional: Version specific test plan
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Integration/Test is not part of the simultaneous release, so the same suites are
+used for testing all supported ODL versions. There may be API changes between
+different releases of ODL, which may require different logic in your Robot
+tests. If the difference is small, it is recommended to act upon value of
+ODL_STREAM variable (e.g. "beryllium", "boron", "carbon", etc).
+
+If the difference is big, you may want to use different list of suites in
+testplan. One way is to define separate jobs with different functionality names.
+But the more convenient way is to define stream-specific testplan. For example::
+
+ vim test/csit/testplans/$project-$functionality-boron.txt
+
+would contain a list of suites for testing Boron, while
+$project-$functionality.txt would still contain the default list (used for
+streams without stream specific testplans).
+
+Optional: Create a script or config plan
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Sometimes the environment prepared by scripts in releng/builder is not suitable
+as is, and there are changes to be done before controller is installed (script
+plan) or before it is started (config plan). You may create as many bash scripts
+as you need in test/csit/scripts/ and then list them in the scriplans or
+configplans folder::
+
+ vim test/csit/scriptplans/$project-$functionality.txt
+
+Save and push changes
+^^^^^^^^^^^^^^^^^^^^^
+Add the changes and push them in the integration/test repo::
+
+ git add -A
+ git commit -s
+ git push
+
+Create system test job
+----------------------
+Download RelEng Builder repository::
+
+ git clone ssh://${USERNAME}@git.opendaylight.org:29418/releng/builder
+ cd builder
+
+Follow the instructions in pulling-and-pushing-the-code_ to know more about
+pulling and pushing code.
+
+Create a new file and modify the values according to your project::
+
+ vim jjb/$project/$project-csit-$functionality.yaml
+
+It should look like this::
+
+ ---
+ - project:
+ name: openflowplugin-csit-flow-services
+ jobs:
+ - '{project}-csit-1node-{functionality}-{install}-{stream}'
+
+ # The project name
+ project: 'openflowplugin'
+
+ # The functionality under test
+ functionality: 'flow-services'
+
+ # Project branches
+ stream:
+ - carbon:
+ branch: 'master'
+ jre: 'openjdk8'
+
+ install:
+ - only:
+ scope: 'only'
+ - all:
+ scope: 'all'
+
+ # Features to install
+ install-features: >
+ odl-openflowplugin-flow-services-ui,
+ odl-openflowplugin-app-bulk-o-matic
+
+ # Robot custom options
+ robot-options: '-v ODL_OF_PLUGIN:lithium'
+
+Explanation:
+
+* name: give some name like $project-csit-$functionality.
+* jobs: replace 1node by 3node if your test is develop for 3node cluster.
+* project: set your your project name here (e.g. openflowplugin).
+* functionality: set the functionality you want to test (e.g. flow-services).
+ Note this has also to match the robot test plan name you defined in earlier
+ `section <Create a test plan_>`_ (e.g. openflowplugin-flow-services.txt)
+* stream: list the project branches you are going to generate system test. Only
+ last branch if the project is new.
+* install: this specifies controller installation, 'only' means only features in
+ install-features will be installed, 'all' means all compatible features will
+ be installed on top (multi-project features test).
+* install-features: list of features you want to install in controller separated
+ by comma.
+* robot-options: robot option you want to pass to the test separated by space.
+
+Save the changes and exit editor.
+
+Optional: Change default tools image
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+By default a system test spins a tools VM that can be used to run some test tool
+like mininet, netconf tool, BGP simulator, etc. The default values are listed
+below and you only need to specify them if you are changing something, for
+example "tools_system_count: 0" will skip the tools VM if you do not need it.
+For a list of available images see images-list_::
+
+ ---
+ - project:
+ name: openflowplugin-csit-flow-services
+ jobs:
+ - '{project}-csit-1node-{functionality}-{install}-{stream}'
+
+ # The project name
+ project: 'openflowplugin'
+
+ # The functionality under test
+ functionality: 'flow-services'
+
+ # Project branches
+ stream:
+ - carbon:
+ branch: 'master'
+ jre: 'openjdk8'
+
+ install:
+ - only:
+ scope: 'only'
+ - all:
+ scope: 'all'
+
+ # Job images (optional)
+ tools_system_count: 1
+ tools_system_flavor: 2 GB General Purpose v1
+ tools_system_image: Ubuntu 14.04 - mininet - 20170210-0439
+
+ # Features to install
+ install-features: >
+ odl-openflowplugin-flow-services-ui,
+ odl-openflowplugin-app-bulk-o-matic
+
+ # Robot custom options
+ robot-options: '-v ODL_OF_PLUGIN:lithium'
+
+
+Optional: Plot a graph from your job
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Scalability and peformance tests not only PASS/FAIL but most important they
+provide a number or value we want to plot in a graph and track over different
+builds.
+
+For that you can add the plot configuration like in this example below::
+
+ ---
+ - project:
+ name: openflowplugin-csit-cbench-performance
+ jobs:
+ - '{project}-csit-1node-{functionality}-{install}-{stream}'
+
+ # The project name
+ project: 'openflowplugin'
+
+ # The functionality under test
+ functionality: 'cbench-performance'
+
+ # Project branches
+ stream:
+ - carbon:
+ branch: 'master'
+ jre: 'openjdk8'
+ - boron:
+ branch: 'stable/boron'
+ jre: 'openjdk8'
+
+ install:
+ - only:
+ scope: 'only'
+
+ # Features to install
+ install-features: 'odl-openflowplugin-flow-services-ui,odl-openflowplugin-drop-test'
+
+ # Robot custom options
+ robot-options: '-v throughput_threshold:20000 -v latency_threshold:5000'
+
+ # Plot Info
+ 01-plot-title: 'Throughput Mode'
+ 01-plot-yaxis: 'flow_mods/sec'
+ 01-plot-group: 'Cbench Performance'
+ 01-plot-data-file: 'throughput.csv'
+ 02-plot-title: 'Latency Mode'
+ 02-plot-yaxis: 'flow_mods/sec'
+ 02-plot-group: 'Cbench Performance'
+ 02-plot-data-file: 'latency.csv'
+
+Explanation:
+
+* There are up to 10 plots per job and every plot can track different values,
+ for example max, min, average recorded in a csv file. In the example above you
+ can skip the 02-* lines if you do not use second plot.
+* plot-title: title for your plot.
+* plot-yaxis: your measurement (xaxis is build # so no need to fill).
+* plot-group: just a label, use the same in case you have 2 plots.
+* plot-data-file: this is the csv file generated by robot framework and contains
+ the values to plot. Examples can be found in openflow-performance_.
+
+Optional: Add Patch Test Job to verify project patches
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+With the steps above your new csit job will run daily on latest generated
+distribution. There is one more extra and optional step if you also want to run
+your system test to verify patches in your project.
+
+The patch test is triggered in gerrit using the keyword::
+
+ test-$project-$feature
+
+The job will:
+
+* Build the gerrit patch.
+* Create a distribution containing the patch.
+* Trigger some system test (csit) that already exists and you specify with the
+ $feature definition below.
+
+Create $project-patch-test.yaml file in your jjb folder::
+
+ vim jjb/$project/$project-patch-test-jobs.yaml
+
+Fill the information as below::
+
+ ---
+ - project:
+ name: openflowplugin-patch-test
+ jobs:
+ - '{project}-patch-test-{feature}-{stream}'
+
+ # The project name
+ project: 'openflowplugin'
+
+ # Project branches
+ stream:
+ - carbon:
+ branch: 'master'
+ jdk: 'openjdk8'
+ - boron:
+ branch: 'stable/boron'
+ jdk: 'openjdk8'
+
+ feature:
+ - core:
+ csit-list: >
+ openflowplugin-csit-1node-flow-services-only-{stream},
+ openflowplugin-csit-1node-flow-services-all-{stream},
+ openflowplugin-csit-1node-scalability-only-{stream},
+ openflowplugin-csit-1node-cbench-performance-only-{stream},
+ openflowplugin-csit-1node-config-performance-only-{stream},
+ openflowplugin-csit-3node-clustering-only-{stream}
+
+ - netvirt:
+ csit-list: >
+ netvirt-csit-1node-openstack-mitaka-gate-transparent-{stream}
+
+ - cluster-netvirt:
+ csit-list: >
+ netvirt-csit-3node-openstack-mitaka-gate-transparent-{stream}
+
+Explanation:
+
+* name: give some name like $project-patch-test.
+* project: set your your project name here (e.g. openflowplugin).
+* stream: list the project branches you are going to generate system test. Only
+ last branch if the project is new.
+* feature: you can group system tests in features. Note there is a predefined
+ feature -all- that triggers all features together.
+* Fill the csit-list with all the system test jobs you want to run to verify a
+ feature.
+
+Debug System Test
+-----------------
+Before pushing your system test job into jenkins-releng_, it is recommended to
+debug the job as well as the you system test code in the sandbox. To do that:
+
+* Set up sandbox access using jenkins-sandbox-install_ instruction.
+* Push you new csit job to sandbox::
+
+ jenkins-jobs --conf jenkins.ini update jjb/ $project-csit-1node-$functionality-only-$branch
+
+* Open your job in jenkins-sandbox_ and start a build replacing the PATCHREFSPEC
+ parameter by your int/test patch REFSPEC (e.g. refs/changes/85/23185/1). you
+ can find this info in gerrit top right corner 'Download' button.
+* Update the PATCHREFSPEC parameter every time you push a new patchset in the
+ int/test repository.
+
+Optional: Debug VM issues in sandbox
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+In case of problems with the test VMs, you can easily debug these issues in the
+sandbox by adding the following lines in a Jenkins shell window::
+
+ cat > ${WORKSPACE}/debug-script.sh <<EOF
+
+ <<put your debug shell script here>>
+
+ EOF
+ scp ${WORKSPACE}/debug-script.sh ${TOOLS_SYSTEM_IP}:/tmp
+ ssh ${TOOLS_SYSTEM_IP} 'sudo bash /tmp/debug-script.sh'
+
+Note this will run a self-made debug script with sudo access in a VM of your
+choice. In the example above you debug on the tools VM (TOOLS_SYSTEM_IP),
+use ODL_SYSTEM_IP to debug in controller VM.
+
+Save and push changes
+^^^^^^^^^^^^^^^^^^^^^
+Once you are happy with your system test, save the changes and push them in the
+releng builder repo::
+
+ git add -A
+ git commit -s
+ git push
+
+.. important::
+
+ If this is your first system test job, it is recommended to add the int/test
+ patch (gerrit link) in the commit message so that committers can merge both
+ the int/test and the releng/builder patches at the same time.
+
+Check system test jobs in Jenkins
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Once your patches are merged your system test can be browsed in jenkins-releng_:
+
+* $project-csit-1node-$functionality-only-$branch -> The single-feature test.
+* $project-csit-1node-$functionality-all-$branch -> The multi-project test.
+* $yourproject-patch-test-$feature-$branch -> Patch test job.
+
+Note that jobs in jenkins-releng_ cannot be reconfigured, only jobs in
+jenkins-sandbox_ can, that is why it is so important for testers to get access
+to sandbox.
+
+Support
+-------
+Integration people are happy to support with questions and recommendations:
+
+* Integration IRC: OpenDaylight channel 'opendaylight-integration
+* Integration Mail: OpenDaylight list 'integration-dev@lists.opendaylight.org'
+
+.. _pulling-and-pushing-the-code: http://docs.opendaylight.org/en/stable-boron/developer-guide/pulling-and-pushing-the-code-from-the-cli.html
+.. _images-list: http://docs.opendaylight.org/en/stable-boron/submodules/releng/builder/docs/jenkins.html#pool-odlpub-hot-heat-orchestration-templates
+.. _openflow-performance: https://git.opendaylight.org/gerrit/gitweb?p=integration/test.git;a=blob;f=csit/suites/openflowplugin/Performance/010_Cbench.robot
+.. _jenkins-releng: https://jenkins.opendaylight.org/releng/
+.. _jenkins-sandbox: https://jenkins.opendaylight.org/sandbox/
+.. _jenkins-sandbox-install: http://docs.opendaylight.org/en/stable-boron/submodules/releng/builder/docs/jenkins.html#jenkins-sandbox
Code Review / integration / test.git / blobdiff