+.. _odl-jenkins:
+
Jenkins
=======
Please note that the combination of a Packer definitions from `vars`, `templates`
and the `provision` scripts is what defines a given minion. For instance, a minion
-may be defined as `centos7-java-builder` which is a combination of Packer OS image
+may be defined as `centos7-builder` which is a combination of Packer OS image
definitions from `vars/centos.json`, Packer template definitions from
-`templates/java-buidler.json` and spinup scripts from `provision/java-builder.sh`.
+`templates/builder.json` and spinup scripts from `provision/builder.sh`.
This combination provides the full definition of the realized minion.
Jenkins starts a minion using the latest image which is built and linked into the
executed Jenkins will finally attach the minion as an actual minion and start
handling jobs on it.
-Pool: ODLRPC
+Flavors
+^^^^^^^
+
+Performance flavors come with dedicated CPUs and are not shared with other
+accounts in the cloud so should ensure consistent performance.
+
+.. list-table:: Flavors
+ :widths: auto
+ :header-rows: 1
+
+ * - Instance Type
+ - CPUs
+ - Memory
+
+ * - odl-standard-1
+ - 1
+ - 4
+
+ * - odl-standard-2
+ - 2
+ - 8
+
+ * - odl-standard-4
+ - 4
+ - 16
+
+ * - odl-standard-8
+ - 8
+ - 32
+
+ * - odl-standard-16
+ - 16
+ - 64
+
+ * - odl-highcpu-2
+ - 2
+ - 2
+
+ * - odl-highcpu-4
+ - 4
+ - 4
+
+ * - odl-highcpu-8
+ - 8
+ - 8
+
+Pool: ODLVEX
^^^^^^^^^^^^
.. raw:: html
<table class="table table-bordered">
<tr class="warning">
- <td><b>Jenkins Labels</b><br/> centos7-java-builder-2c-4g,
- centos7-java-builder-2c-8g, centos7-java-builder-4c-8g,
- centos7-java-builder-8c-8g, centos7-java-builder-4c-16g</td>
- <td><b>Minion Template names</b><br/> centos7-java-builder-2c-4g,
- centos7-java-builder-2c-4g, centos7-java-builder-2c-8g,
- centos7-java-builder-4c-8g, centos7-java-builder-8c-8g,
- centos7-java-builder-4c-16g</td>
+ <td><b>Jenkins Labels</b><br/>
+ centos7-builder-2c-1g,<br/>
+ centos7-builder-2c-2g,<br/>
+ centos7-builder-2c-8g,<br/>
+ centos7-builder-4c-4g,<br/>
+ centos7-builder-8c-8g,<br/>
+ centos7-autorelease-4c-16g
+ </td>
+ <td><b>Minion Template names</b><br/>
+ prd-centos7-builder-2c-1g,<br/>
+ prd-centos7-builder-2c-2g,<br/>
+ prd-centos7-builder-2c-8g,<br/>
+ prd-centos7-builder-4c-4g,<br/>
+ prd-centos7-builder-8c-8g,<br/>
+ prd-centos7-autorelease-4c-16g
<td><b>Packer Template</b><br/>
- releng/builder/packer/templates/java-builder.json</td>
- <td><b>Spinup Script</b><br/>
- releng/builder/jenkins-scripts/builder.sh</td>
+ releng/builder/packer/templates/builder.json</td>
+ <td><b>Playbook</b><br/>
+ releng/builder/packer/common-packer/provision/baseline.yaml</td>
</tr>
<tr>
<td colspan="4">
<td><b>Minion Template names</b><br/> centos7-robot-2c-2g</td>
<td><b>Packer Template</b><br/>
releng/builder/packer/templates/robot.json</td>
- <td><b>Spinup Script</b><br/> releng/builder/jenkins-scripts/robot.sh</td>
+ <td><b>Playbook</b><br/> releng/builder/packer/provision/robot.yaml</td>
</tr>
<tr>
<td colspan="4">
</tr>
<tr class="warning">
- <td><b>Jenkins Labels</b><br/> ubuntu1404-mininet-2c-2g</td>
- <td><b>Minion Template names</b><br/> ubuntu1404-mininet-2c-2g</td>
- <td><b>Packer Template</b><br/>
- releng/builder/packer/teamplates/mininet.json</td>
- <td><b>Spinup Script</b><br/> releng/builder/jenkins-scripts/mininet-ubuntu.sh</td>
+ <td><b>Jenkins Labels</b><br/> ubuntu1604-mininet-ovs-25-2c-2g</td>
+ <td><b>Minion Template names</b><br/> ubuntu1604-mininet-ovs-25-2c-2g</td>
+ <td><b>Packer Template</b><br/> releng/builder/packer/templates/mininet-ovs-2.5.json</td>
+ <td><b>Playbook</b><br/> releng/builder/packer/provision//mininet-ovs-2.5.yaml</td>
</tr>
<tr>
<td colspan="4">
- Basic Ubuntu 14.04 (Trusty) system with ovs 2.0.2 and mininet 2.1.0
+ Basic Ubuntu 16.04 (Xenial) system with ovs 2.5 and mininet 2.2.1
</td>
</tr>
<tr class="warning">
- <td><b>Jenkins Labels</b><br/> ubuntu1404-mininet-ovs-23-2c-2g</td>
- <td><b>Minion Template names</b><br/> ubuntu1404-mininet-ovs-23-2c-2g</td>
- <td><b>Packer Template</b><br/> releng/builder/packer/templates/mininet-ovs-2.3.json</td>
- <td><b>Spinup Script</b><br/> releng/builder/jenkins-scripts/mininet-ubuntu.sh</td>
+ <td><b>Jenkins Labels</b><br/> ubuntu1604-mininet-ovs-26-2c-2g</td>
+ <td><b>Minion Template names</b><br/> ubuntu1604-mininet-ovs-26-2c-2g</td>
+ <td><b>Packer Template</b><br/> releng/builder/packer/templates/mininet-ovs-2.6.json</td>
+ <td><b>Playbook</b><br/> releng/builder/packer/provision//mininet-ovs-2.6.yaml</td>
</tr>
<tr>
<td colspan="4">
- Ubuntu 16.04 (Xenial) system with ovs 2.5 and mininet 2.2.1
+ Ubuntu 16.04 (Xenial) system with ovs 2.6 and mininet 2.2.1
+ </td>
+ </tr>
+
+ <tr class="warning">
+ <td><b>Jenkins Labels</b><br/> ubuntu1604-mininet-ovs-28-2c-2g</td>
+ <td><b>Minion Template names</b><br/> ubuntu1604-mininet-ovs-28-2c-2g</td>
+ <td><b>Packer Template</b><br/> releng/builder/packer/templates/mininet-ovs-2.8.json</td>
+ <td><b>Playbook</b><br/> releng/builder/packer/provision//mininet-ovs-2.8.yaml</td>
+ </tr>
+ <tr>
+ <td colspan="4">
+ Ubuntu 16.04 (Xenial) system with ovs 2.8 and mininet 2.2.1
</td>
</tr>
<td><b>Jenkins Labels</b><br/> centos7-devstack-2c-4g</td>
<td><b>Minion Template names</b><br/> centos7-devstack-2c-4g</td>
<td><b>Packer Template</b><br/> releng/builder/packer/templates/devstack.json</td>
- <td><b>Spinup Script</b><br/> releng/builder/jenkins-scripts/devstack.sh</td>
+ <td><b>Playbook</b><br/> releng/builder/packer/provision/devstack.yaml</td>
</tr>
<tr>
<td colspan="4">
<td><b>Jenkins Labels</b><br/> centos7-docker-2c-4g</td>
<td><b>Minion Template names</b><br/> centos7-docker-2c-4g</td>
<td><b>Packer Template</b><br/> releng/builder/packer/templates/docker.json</td>
- <td><b>Spinup Script</b><br/> releng/builder/jenkins-scripts/docker.sh</td>
+ <td><b>Playbook</b><br/> releng/builder/packer/common-packer/provision/docker.yaml</td>
</tr>
<tr>
<td colspan="4">
</tr>
<tr class="warning">
- <td><b>Jenkins Labels</b><br/> ubuntu1404-gbp-2c-2g</td>
- <td><b>Minion Template names</b><br/> ubuntu1404-gbp-2c-2g</td>
- <td><b>Packer Template</b><br/> releng/builder/packer/templates/gbp.json</td>
- <td><b>Spinup Script</b><br/> releng/builder/jenkins-scripts/ubuntu-docker-ovs.sh</td>
- </tr>
- <tr>
- <td colspan="4">
- Ubuntu 14.04 (Trusty) node with latest OVS and docker installed. Used by Group Based Policy.
- </td>
- </tr>
-
- <tr class="warning">
- <td><b>Jenkins Labels</b><br/> ubuntu1604-gbp-2c-4g</td>
- <td><b>Minion Template names</b><br/> ubuntu1604-gbp-2c-4g</td>
+ <td><b>Jenkins Labels</b><br/> ubuntu1604-gbp-2c-2g</td>
+ <td><b>Minion Template names</b><br/> ubuntu1604-gbp-2c-2g</td>
<td><b>Packer Template</b><br/> releng/builder/packer/templates/gbp.json</td>
- <td><b>Spinup Script</b><br/> releng/builder/jenkins-scripts/ubuntu-docker-ovs.sh</td>
+ <td><b>Spinup Script</b><br/> releng/builder/jenkins-scripts/gpb.sh</td>
</tr>
<tr>
<td colspan="4">
</table>
-Pool: ODLPUB - HOT (Heat Orchestration Templates)
+Pool: ODLVEX - HOT (Heat Orchestration Templates)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
HOT integration enables to spin up integration labs servers for CSIT jobs
`{odl,docker,openstack,tools}_system_image` followed by image name in the
format `<platform> - <template> - <date-stamp>`.
-.. code-block:: yaml
-
- CentOS 7 - docker - 20161031-0802
-
.. include:: cloud-images.rst
Creating Jenkins Jobs
$ cat jjb/requirements.txt
-e git+https://git.openstack.org/openstack-infra/jenkins-job-builder@1.4.0#egg=jenkins-job-builder
+Updating releng/builder repo or global-jjb
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Follow these steps to update the releng/builder repo. The repo uses a submodule from
+a global-jjb repo so that common source can be shared across different projects. This
+requires updating the releng/builder repo periodically to pick up the changes. New
+versions of jjb could also require updating the releng/builder repo. Follow the
+previous steps earlier for updating jenkins-jobs using the
+`builder/jjb/requirements.txt <odl-jjb-requirements.txt_>`_ file. Ensure that the
+version listed in the file is the currently supported version, otherwise install a
+different version or simply upgrade using `pip install --upgrade jenkins-job-builder`.
+
+The example below assumes the user has cloned releng/builder to `~/git/releng/builder`.
+Update the repo, update the submodules and then submit a test to verify it works.
+
+.. code-block:: bash
+
+ cd ~/git/releng/builder
+ git checkout master
+ git pull
+ git submodule update --init --recursive
+ jenkins-jobs --conf jenkins.ini test jjb/ netvirt-csit-1node-openstack-queens-upstream-stateful-fluorine
+
Installing JJB Manually
-----------------------
</td>
</tr>
- <tr class="warning">
- <td><b>Job Template</b><br/>integration-patch-test-{stream}</td>
- <td><b>Gerrit Trigger</b><br/>test-integration</td>
- </tr>
- <tr>
- <td colspan="2">
- </td>
- </tr>
-
<tr class="warning">
<td><b>Job Template</b><br/>integration-patch-test-{stream}</td>
<td><b>Gerrit Trigger</b><br/>test-integration</td>
</li>
</td>
</tr>
+
+ <tr class="warning">
+ <td><b>Job Template</b><br/>integration-multipatch-test-{stream}</td>
+ <td><b>Gerrit Trigger</b><br/>multipatch-build</td>
+ </tr>
+ <tr>
+ <td colspan="2">
+ This job builds a list of patches provided in an specific order, and finally builds
+ a distribution from either provided patch or latest code in branch.
+ For example if someone leaves the following comment in a patch:
+ multipatch-build:controller=61/29761/5:45/29645/6,neutron=51/65551/4,netvirt:59/60259/17
+ the job will checkout controller patch 61/29761/5, cherry-pick 45/29645/6 and build controller,
+ checkout neutron patch 51/65551/4 and build neutron, checkout latest netvirt code,
+ cherry-pick 59/60259/17 and build netvirt, finally it will checkout latest distribution
+ code and build a distribution. The resulting distribution is stored in Nexus and the URL
+ is stored in a variable called BUNDLE_URL visible in the job console.
+ This job also accepts a gerrit topic, for example: multipatch-build:topic=binding-tlc-rpc,
+ in this case the job will find all patches in the topic binding-tlc-rpc for the projects
+ specified in the BUILD_ORDER parameter and will build all projects from the first a patch
+ has been found, for successive projects the branch HEAD is used if no patch is found.
+ The job uses patch numbers to sort patches in the same project.
+ Use multipatch-build-fast (vs multipatch-build) for building projects fast (-Pq).
+ This job should not alter Gerrit votes for a given patch, nor will do anything with the
+ given patch unless the patch is added to the build list.
+ </td>
+ </tr>
+
</table>
Maven Properties
Maven property
<sonar>true</sonar>.
+.. _odl-jenkins-sandbox:
+
Jenkins Sandbox
---------------
-The `jenkins-sandbox`_ instance's purpose is to allow projects to test their JJB
-setups before merging their code over to the RelEng master silo. It is
-configured similarly to the master instance, although it cannot publish
-artifacts or vote in Gerrit.
-
-If your project requires access to the sandbox please open an OpenDaylight
-Helpdesk ticket (<helpdesk@opendaylight.org>) and provide your ODL ID.
-
-Notes Regarding the Sandbox
-^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-* Jobs are automatically deleted every Saturday at 08:00 UTC
-* Committers can login and configure Jenkins jobs in the sandbox directly
- (unlike with the master silo)
-* Sandbox configuration mirrors the master silo when possible
-* Sandbox jobs can NOT upload artifacts to Nexus
-* Sandbox jobs can NOT vote on Gerrit
-
-Configuration
-^^^^^^^^^^^^^
-
-Make sure you have Jenkins Job Builder [properly installed](#jjb_install).
-
-If you do not already have access, open an OpenDaylight Helpdesk ticket
-(<helpdesk@opendaylight.org>) to request access to ODL's sandbox instance.
-Integration/Test (`integration-test-wiki`_) committers have access by default.
-
-JJB reads user-specific configuration from a `jenkins.ini`_. An
-example is provided by releng/builder at `example-jenkins.ini`_.
-
-.. code-block:: bash
-
- # If you don't have RelEng/Builder's repo, clone it
- $ git clone --recursive https://git.opendaylight.org/gerrit/p/releng/builder.git
- # Make a copy of the example JJB config file (in the builder/ directory)
- $ cp jenkins.ini.example jenkins.ini
- # Edit jenkins.ini with your username, API token and ODL's sandbox URL
- $ cat jenkins.ini
- <snip>
- [jenkins]
- user=<your ODL username>
- password=<your ODL Jenkins sandbox API token>
- url=https://jenkins.opendaylight.org/sandbox
- <snip>
-
-To get your API token, `login to the Jenkins **sandbox** instance
-<jenkins-sandbox-login_>`_ (*not
-the main master Jenkins instance, different tokens*), go to your user page (by
-clicking on your username, for example), click "Configure" and then "Show API
-Token".
-
-Manual Method
-^^^^^^^^^^^^^
-
-If you `installed JJB locally into a virtual environment
-<Installing Jenkins Job Builder_>`_,
-you should now activate that virtual environment to access the `jenkins-jobs`
-executable.
-
-.. code-block:: bash
-
- $ workon jjb
- (jjb)$
-
-You'll want to work from the root of the RelEng/Builder repo, and you should
-have your `jenkins.ini` file [properly configured](#sandbox_config).
-
-Testing Jobs
-^^^^^^^^^^^^
-
-It's good practice to use the `test` command to validate your JJB files before
-pushing them.
-
-.. code-block:: bash
-
- jenkins-jobs --conf jenkins.ini test jjb/ <job-name>
-
-If the job you'd like to test is a template with variables in its name, it
-must be manually expanded before use. For example, the commonly used template
-`{project}-csit-verify-1node-{functionality}` might expand to
-`ovsdb-csit-verify-1node-netvirt`.
-
-.. code-block:: bash
-
- jenkins-jobs --conf jenkins.ini test jjb/ ovsdb-csit-verify-1node-netvirt
-
-Successful tests output the XML description of the Jenkins job described by
-the specified JJB job name.
-
-Pushing Jobs
-^^^^^^^^^^^^
-
-Once you've `configured your \`jenkins.ini\` <Configuration_>`_ and `verified your
-JJB jobs <Testing Jobs_>`_ produce valid XML descriptions of Jenkins jobs you
-can push them to the Jenkins sandbox.
-
-.. important::
-
- When pushing with `jenkins-jobs`, a log message with the number
- of jobs you're pushing will be issued, typically to stdout.
- **If the number is greater than 1** (or the number of jobs you
- passed to the command to push) then you are pushing too many
- jobs and should **`ctrl+c` to cancel the upload**. Else you will
- flood the system with jobs.
-
- .. code-block:: bash
-
- INFO:jenkins_jobs.builder:Number of jobs generated: 1
-
- **Failing to provide the final `<job-name>` param will push all
- jobs!**
-
- .. code-block:: bash
-
- # Don't push all jobs by omitting the final param! (ctrl+c to abort)
- jenkins-jobs --conf jenkins.ini update jjb/ <job-name>
-
-Alternatively, you can push a job to the Jenkins sandbox with a special comment in a
-releng/builder gerrit patch. The job will be based off of the code your patch is
-based upon. Meaning, if your patch is changing something related to the job you are
-pushing, those changes will exist in the sandbox job. The format of the comment is::
-
- jjb-deploy <job name>
-
-Running Jobs
-^^^^^^^^^^^^
-
-Once you have your Jenkins job configuration `pushed to the
-Sandbox <Pushing Jobs_>`_ you can trigger it to run.
-
-Find your newly-pushed job on the `Sandbox's web UI <jenkins-sandbox_>`_. Click
-on its name to see the job's details.
-
-Make sure you're `logged in <jenkins-sandbox-login_>`_ to the Sandbox.
-
-Click "Build with Parameters" and then "Build".
-
-Wait for your job to be scheduled and run. Click on the job number to see
-details, including console output.
-
-Make changes to your JJB configuration, re-test, re-push and re-run until
-your job is ready.
+URL: https://jenkins.opendaylight.org/sandbox
+Jenkins Sandbox documentation can be found in the
+:doc:`LF Jenkins Sandbox Guide <lfdocs:jenkins-sandbox>`.
.. _example-jenkins.ini: https://git.opendaylight.org/gerrit/gitweb?p=releng/builder.git;a=blob;f=jenkins.ini.example
.. _integration-test-wiki: https://wiki.opendaylight.org/view/Integration/Test
.. _jenkins-master: https://jenkins.opendaylight.org/releng
-.. _jenkins-sandbox: https://jenkins.opendaylight.org/sandbox
-.. _jenkins-sandbox-login: https://jenkins.opendaylight.org/sandbox/login
.. _jenkins.ini: http://docs.openstack.org/infra/jenkins-job-builder/execution.html#configuration-file
.. _jjb-autoupdate-project.py: https://git.opendaylight.org/gerrit/gitweb?p=releng/builder.git;a=blob;f=scripts/jjb-autoupdate-project.py
.. _jjb-docs: http://ci.openstack.org/jenkins-job-builder/
Code Review / releng / builder.git / blobdiff