X-Git-Url: https://git.opendaylight.org/gerrit/gitweb?a=blobdiff_plain;f=docs%2Fjenkins.rst;h=e8a0c1962de7d60a661f0ac4334d9953beb9654f;hb=5d51a149c5352941444cb4e28492959b7763a07f;hp=40057f03b9e2b3d31dc6dc648b157731515628ef;hpb=e9379cbbf67ee9f58a24e437782d92534d8b5449;p=releng%2Fbuilder.git

diff --git a/docs/jenkins.rst b/docs/jenkins.rst
index 40057f03b..e8a0c1962 100644
--- a/docs/jenkins.rst
+++ b/docs/jenkins.rst
@@ -1,3 +1,5 @@
+.. _odl-jenkins:
+
 Jenkins
 =======
 
@@ -169,9 +171,9 @@ triggered by verify-packer and merge-packer jobs.
 
 Please note that the combination of a Packer definitions from `vars`, `templates`
 and the `provision` scripts is what defines a given minion. For instance, a minion
-may be defined as `centos7-java-builder` which is a combination of Packer OS image
+may be defined as `centos7-builder` which is a combination of Packer OS image
 definitions from `vars/centos.json`, Packer template definitions from
-`templates/java-buidler.json` and spinup scripts from `provision/java-builder.sh`.
+`templates/builder.json` and spinup scripts from `provision/builder.sh`.
 This combination provides the full definition of the realized minion.
 
 Jenkins starts a minion using the latest image which is built and linked into the
@@ -188,24 +190,78 @@ time for every job which runs on the image. After all of these scripts have
 executed Jenkins will finally attach the minion as an actual minion and start
 handling jobs on it.
 
-Pool: ODLRPC
+Flavors
+^^^^^^^
+
+Performance flavors come with dedicated CPUs and are not shared with other
+accounts in the cloud so should ensure consistent performance.
+
+.. list-table:: Flavors
+   :widths: auto
+   :header-rows: 1
+
+   * - Instance Type
+     - CPUs
+     - Memory
+
+   * - odl-standard-1
+     - 1
+     - 4
+
+   * - odl-standard-2
+     - 2
+     - 8
+
+   * - odl-standard-4
+     - 4
+     - 16
+
+   * - odl-standard-8
+     - 8
+     - 32
+
+   * - odl-standard-16
+     - 16
+     - 64
+
+   * - odl-highcpu-2
+     - 2
+     - 2
+
+   * - odl-highcpu-4
+     - 4
+     - 4
+
+   * - odl-highcpu-8
+     - 8
+     - 8
+
+Pool: ODLVEX
 ^^^^^^^^^^^^
 
 .. raw:: html
 
     <table class="table table-bordered">
       <tr class="warning">
-        <td><b>Jenkins Labels</b><br/> centos7-java-builder-2c-4g,
-          centos7-java-builder-2c-8g, centos7-java-builder-4c-8g,
-          centos7-java-builder-8c-8g, centos7-java-builder-4c-16g</td>
-        <td><b>Minion Template names</b><br/> centos7-java-builder-2c-4g,
-          centos7-java-builder-2c-4g, centos7-java-builder-2c-8g,
-          centos7-java-builder-4c-8g, centos7-java-builder-8c-8g,
-          centos7-java-builder-4c-16g</td>
+        <td><b>Jenkins Labels</b><br/>
+          centos7-builder-2c-1g,<br/>
+          centos7-builder-2c-2g,<br/>
+          centos7-builder-2c-8g,<br/>
+          centos7-builder-4c-4g,<br/>
+          centos7-builder-8c-8g,<br/>
+          centos7-autorelease-4c-16g
+        </td>
+        <td><b>Minion Template names</b><br/>
+          prd-centos7-builder-2c-1g,<br/>
+          prd-centos7-builder-2c-2g,<br/>
+          prd-centos7-builder-2c-8g,<br/>
+          prd-centos7-builder-4c-4g,<br/>
+          prd-centos7-builder-8c-8g,<br/>
+          prd-centos7-autorelease-4c-16g
         <td><b>Packer Template</b><br/>
-        releng/builder/packer/templates/java-builder.json</td>
-        <td><b>Spinup Script</b><br/>
-        releng/builder/jenkins-scripts/builder.sh</td>
+        releng/builder/packer/templates/builder.json</td>
+        <td><b>Playbook</b><br/>
+        releng/builder/packer/common-packer/provision/baseline.yaml</td>
       </tr>
       <tr>
         <td colspan="4">
@@ -222,7 +278,7 @@ Pool: ODLRPC
         <td><b>Minion Template names</b><br/> centos7-robot-2c-2g</td>
         <td><b>Packer Template</b><br/>
         releng/builder/packer/templates/robot.json</td>
-        <td><b>Spinup Script</b><br/> releng/builder/jenkins-scripts/robot.sh</td>
+        <td><b>Playbook</b><br/> releng/builder/packer/provision/robot.yaml</td>
       </tr>
       <tr>
         <td colspan="4">
@@ -236,27 +292,38 @@ Pool: ODLRPC
       </tr>
 
       <tr class="warning">
-        <td><b>Jenkins Labels</b><br/> ubuntu1404-mininet-2c-2g</td>
-        <td><b>Minion Template names</b><br/> ubuntu1404-mininet-2c-2g</td>
-        <td><b>Packer Template</b><br/>
-        releng/builder/packer/teamplates/mininet.json</td>
-        <td><b>Spinup Script</b><br/> releng/builder/jenkins-scripts/mininet-ubuntu.sh</td>
+        <td><b>Jenkins Labels</b><br/> ubuntu1604-mininet-ovs-25-2c-2g</td>
+        <td><b>Minion Template names</b><br/> ubuntu1604-mininet-ovs-25-2c-2g</td>
+        <td><b>Packer Template</b><br/> releng/builder/packer/templates/mininet-ovs-2.5.json</td>
+        <td><b>Playbook</b><br/> releng/builder/packer/provision//mininet-ovs-2.5.yaml</td>
       </tr>
       <tr>
         <td colspan="4">
-          Basic Ubuntu 14.04 (Trusty) system with ovs 2.0.2 and mininet 2.1.0
+          Basic Ubuntu 16.04 (Xenial) system with ovs 2.5 and mininet 2.2.1
         </td>
       </tr>
 
       <tr class="warning">
-        <td><b>Jenkins Labels</b><br/> ubuntu1404-mininet-ovs-23-2c-2g</td>
-        <td><b>Minion Template names</b><br/> ubuntu1404-mininet-ovs-23-2c-2g</td>
-        <td><b>Packer Template</b><br/> releng/builder/packer/templates/mininet-ovs-2.3.json</td>
-        <td><b>Spinup Script</b><br/> releng/builder/jenkins-scripts/mininet-ubuntu.sh</td>
+        <td><b>Jenkins Labels</b><br/> ubuntu1604-mininet-ovs-26-2c-2g</td>
+        <td><b>Minion Template names</b><br/> ubuntu1604-mininet-ovs-26-2c-2g</td>
+        <td><b>Packer Template</b><br/> releng/builder/packer/templates/mininet-ovs-2.6.json</td>
+        <td><b>Playbook</b><br/> releng/builder/packer/provision//mininet-ovs-2.6.yaml</td>
       </tr>
       <tr>
         <td colspan="4">
-          Ubuntu 16.04 (Xenial) system with ovs 2.5 and mininet 2.2.1
+          Ubuntu 16.04 (Xenial) system with ovs 2.6 and mininet 2.2.1
+        </td>
+      </tr>
+
+      <tr class="warning">
+        <td><b>Jenkins Labels</b><br/> ubuntu1604-mininet-ovs-28-2c-2g</td>
+        <td><b>Minion Template names</b><br/> ubuntu1604-mininet-ovs-28-2c-2g</td>
+        <td><b>Packer Template</b><br/> releng/builder/packer/templates/mininet-ovs-2.8.json</td>
+        <td><b>Playbook</b><br/> releng/builder/packer/provision//mininet-ovs-2.8.yaml</td>
+      </tr>
+      <tr>
+        <td colspan="4">
+          Ubuntu 16.04 (Xenial) system with ovs 2.8 and mininet 2.2.1
         </td>
       </tr>
 
@@ -264,7 +331,7 @@ Pool: ODLRPC
         <td><b>Jenkins Labels</b><br/> centos7-devstack-2c-4g</td>
         <td><b>Minion Template names</b><br/> centos7-devstack-2c-4g</td>
         <td><b>Packer Template</b><br/> releng/builder/packer/templates/devstack.json</td>
-        <td><b>Spinup Script</b><br/> releng/builder/jenkins-scripts/devstack.sh</td>
+        <td><b>Playbook</b><br/> releng/builder/packer/provision/devstack.yaml</td>
       </tr>
       <tr>
         <td colspan="4">
@@ -279,7 +346,7 @@ Pool: ODLRPC
         <td><b>Jenkins Labels</b><br/> centos7-docker-2c-4g</td>
         <td><b>Minion Template names</b><br/> centos7-docker-2c-4g</td>
         <td><b>Packer Template</b><br/> releng/builder/packer/templates/docker.json</td>
-        <td><b>Spinup Script</b><br/> releng/builder/jenkins-scripts/docker.sh</td>
+        <td><b>Playbook</b><br/> releng/builder/packer/common-packer/provision/docker.yaml</td>
       </tr>
       <tr>
         <td colspan="4">
@@ -291,22 +358,10 @@ Pool: ODLRPC
       </tr>
 
       <tr class="warning">
-        <td><b>Jenkins Labels</b><br/> ubuntu1404-gbp-2c-2g</td>
-        <td><b>Minion Template names</b><br/> ubuntu1404-gbp-2c-2g</td>
+        <td><b>Jenkins Labels</b><br/> ubuntu1604-gbp-2c-2g</td>
+        <td><b>Minion Template names</b><br/> ubuntu1604-gbp-2c-2g</td>
         <td><b>Packer Template</b><br/> releng/builder/packer/templates/gbp.json</td>
-        <td><b>Spinup Script</b><br/> releng/builder/jenkins-scripts/ubuntu-docker-ovs.sh</td>
-      </tr>
-      <tr>
-        <td colspan="4">
-          Ubuntu 14.04 (Trusty) node with latest OVS and docker installed. Used by Group Based Policy.
-        </td>
-      </tr>
-
-      <tr class="warning">
-        <td><b>Jenkins Labels</b><br/> ubuntu1604-gbp-2c-4g</td>
-        <td><b>Minion Template names</b><br/> ubuntu1604-gbp-2c-4g</td>
-        <td><b>Packer Template</b><br/> releng/builder/packer/templates/gbp.json</td>
-        <td><b>Spinup Script</b><br/> releng/builder/jenkins-scripts/ubuntu-docker-ovs.sh</td>
+        <td><b>Spinup Script</b><br/> releng/builder/jenkins-scripts/gpb.sh</td>
       </tr>
       <tr>
         <td colspan="4">
@@ -316,7 +371,7 @@ Pool: ODLRPC
 
     </table>
 
-Pool: ODLPUB - HOT (Heat Orchestration Templates)
+Pool: ODLVEX - HOT (Heat Orchestration Templates)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 HOT integration enables to spin up integration labs servers for CSIT jobs
@@ -325,10 +380,6 @@ on the project specific job templates using the variable
 `{odl,docker,openstack,tools}_system_image` followed by image name in the
 format `<platform> - <template> - <date-stamp>`.
 
-.. code-block:: yaml
-
-    CentOS 7 - docker - 20161031-0802
-
 .. include:: cloud-images.rst
 
 Creating Jenkins Jobs
@@ -475,6 +526,29 @@ To install from a tag, like 1.4.0:
     $ cat jjb/requirements.txt
     -e git+https://git.openstack.org/openstack-infra/jenkins-job-builder@1.4.0#egg=jenkins-job-builder
 
+Updating releng/builder repo or global-jjb
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Follow these steps to update the releng/builder repo. The repo uses a submodule from
+a global-jjb repo so that common source can be shared across different projects. This
+requires updating the releng/builder repo periodically to pick up the changes. New
+versions of jjb could also require updating the releng/builder repo. Follow the
+previous steps earlier for updating jenkins-jobs using the
+`builder/jjb/requirements.txt <odl-jjb-requirements.txt_>`_ file. Ensure that the
+version listed in the file is the currently supported version, otherwise install a
+different version or simply upgrade using `pip install --upgrade jenkins-job-builder`.
+
+The example below assumes the user has cloned releng/builder to `~/git/releng/builder`.
+Update the repo, update the submodules and then submit a test to verify it works.
+
+.. code-block:: bash
+
+    cd ~/git/releng/builder
+    git checkout master
+    git pull
+    git submodule update --init --recursive
+    jenkins-jobs --conf jenkins.ini test jjb/ netvirt-csit-1node-openstack-queens-upstream-stateful-fluorine
+
 Installing JJB Manually
 -----------------------
 
@@ -683,15 +757,6 @@ TODO: Document test-{project}-{feature} and test-{project}-all.
         </td>
       </tr>
 
-      <tr class="warning">
-        <td><b>Job Template</b><br/>integration-patch-test-{stream}</td>
-        <td><b>Gerrit Trigger</b><br/>test-integration</td>
-      </tr>
-      <tr>
-        <td colspan="2">
-        </td>
-      </tr>
-
       <tr class="warning">
         <td><b>Job Template</b><br/>integration-patch-test-{stream}</td>
         <td><b>Gerrit Trigger</b><br/>test-integration</td>
@@ -723,6 +788,33 @@ TODO: Document test-{project}-{feature} and test-{project}-all.
             </li>
         </td>
       </tr>
+
+      <tr class="warning">
+        <td><b>Job Template</b><br/>integration-multipatch-test-{stream}</td>
+        <td><b>Gerrit Trigger</b><br/>multipatch-build</td>
+      </tr>
+      <tr>
+        <td colspan="2">
+          This job builds a list of patches provided in an specific order, and finally builds
+          a distribution from either provided patch or latest code in branch.
+          For example if someone leaves the following comment in a patch:
+          multipatch-build:controller=61/29761/5:45/29645/6,neutron=51/65551/4,netvirt:59/60259/17
+          the job will checkout controller patch 61/29761/5, cherry-pick 45/29645/6 and build controller,
+          checkout neutron patch 51/65551/4 and build neutron, checkout latest netvirt code,
+          cherry-pick 59/60259/17 and build netvirt, finally it will checkout latest distribution
+          code and build a distribution. The resulting distribution is stored in Nexus and the URL
+          is stored in a variable called BUNDLE_URL visible in the job console.
+          This job also accepts a gerrit topic, for example: multipatch-build:topic=binding-tlc-rpc,
+          in this case the job will find all patches in the topic binding-tlc-rpc for the projects
+          specified in the BUILD_ORDER parameter and will build all projects from the first a patch
+          has been found, for successive projects the branch HEAD is used if no patch is found.
+          The job uses patch numbers to sort patches in the same project.
+          Use multipatch-build-fast (vs multipatch-build) for building projects fast (-Pq).
+          This job should not alter Gerrit votes for a given patch, nor will do anything with the
+          given patch unless the patch is added to the build list.
+        </td>
+      </tr>
+
     </table>
 
 Maven Properties
@@ -743,151 +835,19 @@ wish to run in your project.
                 Maven property
                 <sonar>true</sonar>.
 
+.. _odl-jenkins-sandbox:
+
 Jenkins Sandbox
 ---------------
 
-The `jenkins-sandbox`_ instance's purpose is to allow projects to test their JJB
-setups before merging their code over to the RelEng master silo. It is
-configured similarly to the master instance, although it cannot publish
-artifacts or vote in Gerrit.
-
-If your project requires access to the sandbox please open an OpenDaylight
-Helpdesk ticket (<helpdesk@opendaylight.org>) and provide your ODL ID.
-
-Notes Regarding the Sandbox
-^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-* Jobs are automatically deleted every Saturday at 08:00 UTC
-* Committers can login and configure Jenkins jobs in the sandbox directly
-  (unlike with the master silo)
-* Sandbox configuration mirrors the master silo when possible
-* Sandbox jobs can NOT upload artifacts to Nexus
-* Sandbox jobs can NOT vote on Gerrit
-
-Configuration
-^^^^^^^^^^^^^
-
-Make sure you have Jenkins Job Builder [properly installed](#jjb_install).
-
-If you do not already have access, open an OpenDaylight Helpdesk ticket
-(<helpdesk@opendaylight.org>) to request access to ODL's sandbox instance.
-Integration/Test (`integration-test-wiki`_) committers have access by default.
-
-JJB reads user-specific configuration from a `jenkins.ini`_. An
-example is provided by releng/builder at `example-jenkins.ini`_.
-
-.. code-block:: bash
-
-    # If you don't have RelEng/Builder's repo, clone it
-    $ git clone --recursive https://git.opendaylight.org/gerrit/p/releng/builder.git
-    # Make a copy of the example JJB config file (in the builder/ directory)
-    $ cp jenkins.ini.example jenkins.ini
-    # Edit jenkins.ini with your username, API token and ODL's sandbox URL
-    $ cat jenkins.ini
-    <snip>
-    [jenkins]
-    user=<your ODL username>
-    password=<your ODL Jenkins sandbox API token>
-    url=https://jenkins.opendaylight.org/sandbox
-    <snip>
-
-To get your API token, `login to the Jenkins **sandbox** instance
-<jenkins-sandbox-login_>`_ (*not
-the main master Jenkins instance, different tokens*), go to your user page (by
-clicking on your username, for example), click "Configure" and then "Show API
-Token".
-
-Manual Method
-^^^^^^^^^^^^^
-
-If you `installed JJB locally into a virtual environment
-<Installing Jenkins Job Builder_>`_,
-you should now activate that virtual environment to access the `jenkins-jobs`
-executable.
-
-.. code-block:: bash
-
-    $ workon jjb
-    (jjb)$
-
-You'll want to work from the root of the RelEng/Builder repo, and you should
-have your `jenkins.ini` file [properly configured](#sandbox_config).
-
-Testing Jobs
-^^^^^^^^^^^^
-
-It's good practice to use the `test` command to validate your JJB files before
-pushing them.
-
-.. code-block:: bash
-
-    jenkins-jobs --conf jenkins.ini test jjb/ <job-name>
-
-If the job you'd like to test is a template with variables in its name, it
-must be manually expanded before use. For example, the commonly used template
-`{project}-csit-verify-1node-{functionality}` might expand to
-`ovsdb-csit-verify-1node-netvirt`.
-
-.. code-block:: bash
-
-    jenkins-jobs --conf jenkins.ini test jjb/ ovsdb-csit-verify-1node-netvirt
-
-Successful tests output the XML description of the Jenkins job described by
-the specified JJB job name.
-
-Pushing Jobs
-^^^^^^^^^^^^
-
-Once you've `configured your \`jenkins.ini\` <Configuration_>`_ and `verified your
-JJB jobs <Testing Jobs_>`_ produce valid XML descriptions of Jenkins jobs you
-can push them to the Jenkins sandbox.
-
-.. important::
-
-    When pushing with `jenkins-jobs`, a log message with the number
-    of jobs you're pushing will be issued, typically to stdout.
-    **If the number is greater than 1** (or the number of jobs you
-    passed to the command to push) then you are pushing too many
-    jobs and should **`ctrl+c` to cancel the upload**. Else you will
-    flood the system with jobs.
-
-    .. code-block:: bash
-
-        INFO:jenkins_jobs.builder:Number of jobs generated:  1
-
-    **Failing to provide the final `<job-name>` param will push all
-    jobs!**
-
-    .. code-block:: bash
-
-        # Don't push all jobs by omitting the final param! (ctrl+c to abort)
-        jenkins-jobs --conf jenkins.ini update jjb/ <job-name>
-
-Running Jobs
-^^^^^^^^^^^^
-
-Once you have your Jenkins job configuration `pushed to the
-Sandbox <Pushing Jobs_>`_ you can trigger it to run.
-
-Find your newly-pushed job on the `Sandbox's web UI <jenkins-sandbox_>`_. Click
-on its name to see the job's details.
-
-Make sure you're `logged in <jenkins-sandbox-login_>`_ to the Sandbox.
-
-Click "Build with Parameters" and then "Build".
-
-Wait for your job to be scheduled and run. Click on the job number to see
-details, including console output.
-
-Make changes to your JJB configuration, re-test, re-push and re-run until
-your job is ready.
+URL: https://jenkins.opendaylight.org/sandbox
 
+Jenkins Sandbox documentation can be found in the
+:doc:`LF Jenkins Sandbox Guide <lfdocs:jenkins-sandbox>`.
 
 .. _example-jenkins.ini: https://git.opendaylight.org/gerrit/gitweb?p=releng/builder.git;a=blob;f=jenkins.ini.example
 .. _integration-test-wiki: https://wiki.opendaylight.org/view/Integration/Test
 .. _jenkins-master: https://jenkins.opendaylight.org/releng
-.. _jenkins-sandbox: https://jenkins.opendaylight.org/sandbox
-.. _jenkins-sandbox-login: https://jenkins.opendaylight.org/sandbox/login
 .. _jenkins.ini: http://docs.openstack.org/infra/jenkins-job-builder/execution.html#configuration-file
 .. _jjb-autoupdate-project.py: https://git.opendaylight.org/gerrit/gitweb?p=releng/builder.git;a=blob;f=scripts/jjb-autoupdate-project.py
 .. _jjb-docs: http://ci.openstack.org/jenkins-job-builder/