X-Git-Url: https://git.opendaylight.org/gerrit/gitweb?a=blobdiff_plain;f=docs%2Fjenkins.rst;h=93c5bdf6eec32550604a5c9699e90c6a157af901;hb=3fabb06fd0e62b6175634002f8f5d841e11d8550;hp=ef38654e8fe4c1e7d138e674daff647d5e6a828c;hpb=d86320c12e060814bf9e8c90106d661b5cf7ede4;p=releng%2Fbuilder.git

diff --git a/docs/jenkins.rst b/docs/jenkins.rst
index ef38654e8..93c5bdf6e 100644
--- a/docs/jenkins.rst
+++ b/docs/jenkins.rst
@@ -1,3 +1,5 @@
+.. _odl-jenkins:
+
 Jenkins
 =======
 
@@ -27,12 +29,18 @@ in the jenkins-master_ silo and this can be achieved by simply creating a
 
 .. code-block:: bash
 
-    git clone https://git.opendaylight.org/gerrit/releng/builder
+    git clone --recursive https://git.opendaylight.org/gerrit/releng/builder
     cd builder
     mkdir jjb/<new-project>
 
+.. note:
+
+    releng/global-jjb is a submodule of releng/builder repository which
+    requires a git submodule update --init or using --recursive with git clone.
+    `releng-global-jjb`_
+
 Where <new-project> should be the same name as your project's git repo in
-Gerrit. So if your project is called "aaa" then create a new jjb/aaa directory.
+Gerrit. If your project is called "aaa" then create a new jjb/aaa directory.
 
 Next we will create <new-project>.yaml as follows:
 
@@ -163,9 +171,9 @@ triggered by verify-packer and merge-packer jobs.
 
 Please note that the combination of a Packer definitions from `vars`, `templates`
 and the `provision` scripts is what defines a given minion. For instance, a minion
-may be defined as `centos7-java-builder` which is a combination of Packer OS image
+may be defined as `centos7-builder` which is a combination of Packer OS image
 definitions from `vars/centos.json`, Packer template definitions from
-`templates/java-buidler.json` and spinup scripts from `provision/java-builder.sh`.
+`templates/builder.json` and spinup scripts from `provision/builder.sh`.
 This combination provides the full definition of the realized minion.
 
 Jenkins starts a minion using the latest image which is built and linked into the
@@ -182,22 +190,76 @@ time for every job which runs on the image. After all of these scripts have
 executed Jenkins will finally attach the minion as an actual minion and start
 handling jobs on it.
 
-Pool: ODLRPC
+Flavors
+^^^^^^^
+
+Performance flavors come with dedicated CPUs and are not shared with other
+accounts in the cloud so should ensure consistent performance.
+
+.. list-table:: Flavors
+   :widths: auto
+   :header-rows: 1
+
+   * - Instance Type
+     - CPUs
+     - Memory
+
+   * - odl-standard-1
+     - 1
+     - 4
+
+   * - odl-standard-2
+     - 2
+     - 8
+
+   * - odl-standard-4
+     - 4
+     - 16
+
+   * - odl-standard-8
+     - 8
+     - 32
+
+   * - odl-standard-16
+     - 16
+     - 64
+
+   * - odl-highcpu-2
+     - 2
+     - 2
+
+   * - odl-highcpu-4
+     - 4
+     - 4
+
+   * - odl-highcpu-8
+     - 8
+     - 8
+
+Pool: ODLVEX
 ^^^^^^^^^^^^
 
 .. raw:: html
 
     <table class="table table-bordered">
       <tr class="warning">
-        <td><b>Jenkins Labels</b><br/> centos7-java-builder-2c-4g,
-          centos7-java-builder-2c-8g, centos7-java-builder-4c-8g,
-          centos7-java-builder-8c-8g, centos7-java-builder-4c-16g</td>
-        <td><b>Minion Template names</b><br/> centos7-java-builder-2c-4g,
-          centos7-java-builder-2c-4g, centos7-java-builder-2c-8g,
-          centos7-java-builder-4c-8g, centos7-java-builder-8c-8g,
-          centos7-java-builder-4c-16g</td>
+        <td><b>Jenkins Labels</b><br/>
+          centos7-builder-2c-1g,<br/>
+          centos7-builder-2c-2g,<br/>
+          centos7-builder-2c-8g,<br/>
+          centos7-builder-4c-4g,<br/>
+          centos7-builder-8c-8g,<br/>
+          centos7-autorelease-4c-16g
+        </td>
+        <td><b>Minion Template names</b><br/>
+          prd-centos7-builder-2c-1g,<br/>
+          prd-centos7-builder-2c-2g,<br/>
+          prd-centos7-builder-2c-8g,<br/>
+          prd-centos7-builder-4c-4g,<br/>
+          prd-centos7-builder-8c-8g,<br/>
+          prd-centos7-autorelease-4c-16g
         <td><b>Packer Template</b><br/>
-        releng/builder/packer/templates/java-builder.json</td>
+        releng/builder/packer/templates/builder.json</td>
         <td><b>Spinup Script</b><br/>
         releng/builder/jenkins-scripts/builder.sh</td>
       </tr>
@@ -248,30 +310,6 @@ Pool: ODLRPC
         <td><b>Packer Template</b><br/> releng/builder/packer/templates/mininet-ovs-2.3.json</td>
         <td><b>Spinup Script</b><br/> releng/builder/jenkins-scripts/mininet-ubuntu.sh</td>
       </tr>
-      <tr>
-        <td colspan="4">
-          Ubuntu 14.04 (Trusty) system with ovs 2.3 and mininet 2.2.1
-        </td>
-      </tr>
-
-      <tr class="warning">
-        <td><b>Jenkins Labels</b><br/> ubuntu1404-mininet-ovs-25-2c-2g</td>
-        <td><b>Minion Template names</b><br/> ubuntu1404-mininet-ovs-25-2c-2g</td>
-        <td><b>Packer Template</b><br/> releng/builder/packer/templates/mininet-ovs-2.5.json</td>
-        <td><b>Spinup Script</b><br/> releng/builder/jenkins-scripts/mininet-ubuntu.sh</td>
-      </tr>
-      <tr>
-        <td colspan="4">
-          Ubuntu 14.04 (Trusty) system with ovs 2.5 and mininet 2.2.2
-        </td>
-      </tr>
-
-      <tr class="warning">
-        <td><b>Jenkins Labels</b><br/> ubuntu1604-mininet-ovs-25-2c-4g</td>
-        <td><b>Minion Template names</b><br/> ubuntu1604-mininet-ovs-25-2c-4g</td>
-        <td><b>Packer Template</b><br/> releng/builder/packer/templates/mininet-ovs-2.5.json</td>
-        <td><b>Spinup Script</b><br/> releng/builder/jenkins-scripts/mininet-ubuntu.sh</td>
-      </tr>
       <tr>
         <td colspan="4">
           Ubuntu 16.04 (Xenial) system with ovs 2.5 and mininet 2.2.1
@@ -334,7 +372,7 @@ Pool: ODLRPC
 
     </table>
 
-Pool: ODLPUB - HOT (Heat Orchestration Templates)
+Pool: ODLVEX - HOT (Heat Orchestration Templates)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 HOT integration enables to spin up integration labs servers for CSIT jobs
@@ -343,10 +381,6 @@ on the project specific job templates using the variable
 `{odl,docker,openstack,tools}_system_image` followed by image name in the
 format `<platform> - <template> - <date-stamp>`.
 
-.. code-block:: yaml
-
-    CentOS 7 - docker - 20161031-0802
-
 .. include:: cloud-images.rst
 
 Creating Jenkins Jobs
@@ -369,16 +403,11 @@ need to use the `jenkins-jobs` executable to translate a set of jobs into
 their XML descriptions and upload them to the sandbox Jenkins server.
 
 We document `installing <Installing Jenkins Job Builder_>`_ `jenkins-jobs`
-below. We also provide
-a `pre-built Docker image <jjb-docker_>`_ with `jenkins-jobs` already installed.
+below.
 
 Installing Jenkins Job Builder
 ------------------------------
 
-For users who aren't already experienced with Docker or otherwise don't want
-to use our `pre-built JJB Docker image <jjb-docker_>`_, installing JJB into a
-virtual environment is an equally good option.
-
 We recommend using `pip <Installing JJB using pip_>`_ to assist with JJB
 installs, but we
 also document `installing from a git repository manually
@@ -401,8 +430,8 @@ installs, we recommend using `Python Virtual Environments <python-virtualenv_>`_
 to manage JJB and its
 Python dependencies. The `python-virtualenvwrapper`_ tool can help you do so.
 
-There are good docs for installing `python-virtualenvwrapper`_. On Linux systems
-with pip (typical), they amount to:
+Documentation is available for installing `python-virtualenvwrapper`_. On Linux
+systems with pip (typical), they amount to:
 
 .. code-block:: bash
 
@@ -456,7 +485,7 @@ First, clone the latest version of the `releng-builder-repo`_.
 
 .. code-block:: bash
 
-    $ git clone https://git.opendaylight.org/gerrit/p/releng/builder.git
+    $ git clone --recursive https://git.opendaylight.org/gerrit/p/releng/builder.git
 
 Before actually installing JJB and its dependencies, make sure you've `created
 and activated <Virtual Environments_>`_ a virtual environment for JJB.
@@ -498,6 +527,29 @@ To install from a tag, like 1.4.0:
     $ cat jjb/requirements.txt
     -e git+https://git.openstack.org/openstack-infra/jenkins-job-builder@1.4.0#egg=jenkins-job-builder
 
+Updating releng/builder repo or global-jjb
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Follow these steps to update the releng/builder repo. The repo uses a submodule from
+a global-jjb repo so that common source can be shared across different projects. This
+requires updating the releng/builder repo periodically to pick up the changes. New
+versions of jjb could also require updating the releng/builder repo. Follow the
+previous steps earlier for updating jenkins-jobs using the
+`builder/jjb/requirements.txt <odl-jjb-requirements.txt_>`_ file. Ensure that the
+version listed in the file is the currently supported version, otherwise install a
+different version or simply upgrade using `pip install --upgrade jenkins-job-builder`.
+
+The example below assumes the user has cloned releng/builder to `~/git/releng/builder`.
+Update the repo, update the submodules and then submit a test to verify it works.
+
+.. code-block:: bash
+
+    cd ~/git/releng/builder
+    git checkout master
+    git pull
+    git submodule update --init --recursive
+    jenkins-jobs --conf jenkins.ini test jjb/ netvirt-csit-1node-openstack-queens-upstream-stateful-fluorine
+
 Installing JJB Manually
 -----------------------
 
@@ -550,30 +602,6 @@ To validate that JJB was successfully installed you can run this command:
 
     (jjb)$ jenkins-jobs --version
 
-JJB Docker Image
-----------------
-
-`Docker <docker-docs_>`_ is an open platform used to create virtualized Linux containers
-for shipping self-contained applications. Docker leverages LinuX Containers
-\(LXC\) running on the same operating system as the host machine, whereas a
-traditional VM runs an operating system over the host.
-
-.. code-block:: bash
-
-    docker pull zxiiro/jjb-docker
-    docker run --rm -v ${PWD}:/jjb jjb-docker
-
-This `Dockerfile <jjb-dockerfile_>`_ created the
-`zxiiro/jjb-docker image <jjb-docker_>`_.
-By default it will run:
-
-.. code-block:: bash
-
-    jenkins-jobs test .
-
-You'll need to use the `-v/--volume=[]` parameter to mount a directory
-containing your YAML files, as well as a configured `jenkins.ini` file if you
-wish to upload your jobs to the `Jenkins Sandbox`_.
 
 Jenkins Job Templates
 ---------------------
@@ -730,15 +758,6 @@ TODO: Document test-{project}-{feature} and test-{project}-all.
         </td>
       </tr>
 
-      <tr class="warning">
-        <td><b>Job Template</b><br/>integration-patch-test-{stream}</td>
-        <td><b>Gerrit Trigger</b><br/>test-integration</td>
-      </tr>
-      <tr>
-        <td colspan="2">
-        </td>
-      </tr>
-
       <tr class="warning">
         <td><b>Job Template</b><br/>integration-patch-test-{stream}</td>
         <td><b>Gerrit Trigger</b><br/>test-integration</td>
@@ -770,6 +789,28 @@ TODO: Document test-{project}-{feature} and test-{project}-all.
             </li>
         </td>
       </tr>
+
+      <tr class="warning">
+        <td><b>Job Template</b><br/>integration-multipatch-test-{stream}</td>
+        <td><b>Gerrit Trigger</b><br/>multipatch-build</td>
+      </tr>
+      <tr>
+        <td colspan="2">
+          This job builds a list of patches provided in an specific order, and finally builds
+          a distribution from either provided patch or latest code in branch.
+          For example if someone leaves the following comment in a patch:
+          multipatch-build:controller=61/29761/5:45/29645/6,neutron=51/65551/4,netvirt:59/60259/17
+          the job will checkout controller patch 61/29761/5, cherry-pick 45/29645/6 and build controller,
+          checkout neutron patch 51/65551/4 and build neutron, checkout latest netvirt code,
+          cherry-pick 59/60259/17 and build netvirt, finally it will checkout latest distribution
+          code and build a distribution. The resulting distribution is stored in Nexus and the URL
+          is stored in a variable called BUNDLE_URL visible in the job console.
+          Use multipatch-build-fast (vs multipatch-build) for building projects fast (-Pq).
+          This job should not alter Gerrit votes for a given patch, nor will do anything with the
+          given patch unless the patch is added to the build list.
+        </td>
+      </tr>
+
     </table>
 
 Maven Properties
@@ -790,6 +831,8 @@ wish to run in your project.
                 Maven property
                 <sonar>true</sonar>.
 
+.. _odl-jenkins-sandbox:
+
 Jenkins Sandbox
 ---------------
 
@@ -826,7 +869,7 @@ example is provided by releng/builder at `example-jenkins.ini`_.
 .. code-block:: bash
 
     # If you don't have RelEng/Builder's repo, clone it
-    $ git clone https://git.opendaylight.org/gerrit/p/releng/builder.git
+    $ git clone --recursive https://git.opendaylight.org/gerrit/p/releng/builder.git
     # Make a copy of the example JJB config file (in the builder/ directory)
     $ cp jenkins.ini.example jenkins.ini
     # Edit jenkins.ini with your username, API token and ODL's sandbox URL
@@ -910,6 +953,18 @@ can push them to the Jenkins sandbox.
         # Don't push all jobs by omitting the final param! (ctrl+c to abort)
         jenkins-jobs --conf jenkins.ini update jjb/ <job-name>
 
+Alternatively, you can push a job to the Jenkins sandbox with a special comment in a
+releng/builder gerrit patch. The job will be based off of the code your patch is
+based upon. Meaning, if your patch is changing something related to the job you are
+pushing, those changes will exist in the sandbox job. The format of the comment is::
+
+        jjb-deploy <job name>
+
+.. note::
+
+    Also note that wildcards can be used in <job name> which
+    will expand all jobs that exist for the pattern.
+
 Running Jobs
 ^^^^^^^^^^^^
 
@@ -929,38 +984,7 @@ details, including console output.
 Make changes to your JJB configuration, re-test, re-push and re-run until
 your job is ready.
 
-Docker Method
-^^^^^^^^^^^^^
-
-If `using Docker <JJB Docker image_>`_:
-
-.. code-block:: bash
-
-    # To test
-    docker run --rm -v ${PWD}:/jjb zxiiro/jjb-docker
-
-.. important::
-
-    When pushing with `jenkins-jobs`, a log message with
-    the number of jobs you're pushing will be issued, typically to stdout.
-    **If the number is greater than 1** (or the number of jobs you passed to
-    the command to push) then you are pushing too many jobs and should **`ctrl+c`
-    to cancel the upload**. Else you will flood the system with jobs.
-
-    .. code-block:: bash
-
-          INFO:jenkins_jobs.builder:Number of jobs generated:  1
-
-    **Failing to provide the final `<job-name>` param will push all jobs!**
-
-    .. code-block:: bash
-
-        # To upload jobs to the sandbox
-        # Please ensure that you include a configured jenkins.ini in your volume mount
-        # Making sure not to push more jobs than expected, ctrl+c to abort
-        docker run --rm -v ${PWD}:/jjb zxiiro/jjb-docker jenkins-jobs --conf jenkins.ini update . openflowplugin-csit-periodic-1node-cds-longevity-only-master
 
-.. _docker-docs: https://www.docker.com/whatisdocker/
 .. _example-jenkins.ini: https://git.opendaylight.org/gerrit/gitweb?p=releng/builder.git;a=blob;f=jenkins.ini.example
 .. _integration-test-wiki: https://wiki.opendaylight.org/view/Integration/Test
 .. _jenkins-master: https://jenkins.opendaylight.org/releng
@@ -968,8 +992,6 @@ If `using Docker <JJB Docker image_>`_:
 .. _jenkins-sandbox-login: https://jenkins.opendaylight.org/sandbox/login
 .. _jenkins.ini: http://docs.openstack.org/infra/jenkins-job-builder/execution.html#configuration-file
 .. _jjb-autoupdate-project.py: https://git.opendaylight.org/gerrit/gitweb?p=releng/builder.git;a=blob;f=scripts/jjb-autoupdate-project.py
-.. _jjb-docker: https://hub.docker.com/r/zxiiro/jjb-docker/
-.. _jjb-dockerfile: https://github.com/zxiiro/jjb-docker/blob/master/Dockerfile
 .. _jjb-docs: http://ci.openstack.org/jenkins-job-builder/
 .. _jjb-init-project.py: https://git.opendaylight.org/gerrit/gitweb?p=releng/builder.git;a=blob;f=scripts/jjb-init-project.py
 .. _jjb-repo: https://github.com/openstack-infra/jenkins-job-builder
@@ -983,6 +1005,7 @@ If `using Docker <JJB Docker image_>`_:
 .. _releng-wiki: https://wiki.opendaylight.org/view/RelEng:Main
 .. _releng-builder-gerrit: https://git.opendaylight.org/gerrit/#/admin/projects/releng/builder
 .. _releng-builder-repo: https://git.opendaylight.org/gerrit/gitweb?p=releng%2Fbuilder.git;a=summary
+.. _releng-global-jjb: https://gerrit.linuxfoundation.org/infra/#/q/project:releng/global-jjb
 .. _releng-builder-wiki: https://wiki.opendaylight.org/view/RelEng/Builder
 .. _streams-design-background: https://lists.opendaylight.org/pipermail/release/2015-July/003139.html
 .. _spinup-scripts: https://git.opendaylight.org/gerrit/gitweb?p=releng/builder.git;a=tree;f=jenkins-scripts