1 The [Release Engineering project](https://wiki.opendaylight.org/view/RelEng:Main "Releng:Main")
2 consolidates the Jenkins jobs from project-specific VMs to a single Jenkins
3 server. Each OpenDaylight project will have a tab on the RelEng Jenkins
4 server. The system utilizes
5 [Jenkins Job Builder](http://ci.openstack.org/jenkins-job-builder/ "JJB")
6 \(JJB\) for the creation and management of the Jenkins jobs.
10 https://jenkins.opendaylight.org/releng/
12 The Jenkins Master server is the new home for all project Jenkins jobs. All
13 maintenance and configuration of these jobs must be done via JJB through the
14 RelEng repo ([https://git.opendaylight.org/gerrit/gitweb?p=releng%2Fbuilder.git;a=summary RelEng/Builder gitweb]).
15 Project contributors can no longer edit the Jenkins jobs directly on the
20 The Jenkins jobs are run on build slaves (executors) which are created on an
21 as-needed basis. If no idle build slaves are available a new VM is brought
22 up. This process can take up to 2 minutes. Once the build slave has finished a
23 job, it will remain online for 45 minutes before shutting down. Subsequent
24 jobs will use an idle build slave if available.
26 Our Jenkins master supports many types of dynamic build slaves. If you are
27 creating custom jobs then you will need to have an idea of what type of slaves
28 are available. The following are the current slave types and descriptions.
29 Slave Template Names are needed for jobs that take advantage of multiple
30 slaves as they must be specifically called out by template name instead of
33 # Adding new components to the slaves
35 If your project needs something added to one of the slaves used during build
36 and test you can help us get things added in faster by doing one of the
39 * Submit a patch to releng/builder for the
40 [Jenkins Spinup script](https://git.opendaylight.org/gerrit/gitweb?p=releng/builder.git;a=tree;f=jenkins-scripts;h=69252dd61ece511bd2018039b40e7836a8d49d21;hb=HEAD)
41 that configures your new piece of software.
43 * Submit a patch to releng/builder for the
44 [Vagrant template's bootstrap.sh](https://git.opendaylight.org/gerrit/gitweb?p=releng/builder.git;a=tree;f=vagrant;h=409a2915d48bbdeea9edc811e1661ae17ca28280;hb=HEAD)
45 that configures your new piece of software.
47 Going the first route will be faster in the short term as we can inspect the
48 changes and make test modifications in the sandbox to verify that it works.
50 The second route, however, is better for the community as a whole as it will
51 allow others that utilize our vagrant startups to replicate our systems more
52 closely. It is, however, more time consuming as an image snapshot needs to be
53 created based on the updated vagrant definition before it can be attached to
54 the sandbox for validation testing.
56 In either case, the changes must be validated in the sandbox with tests to
57 make sure that we don't break current jobs but also that the new software
58 features are operating as intended. Once this is done the changes will be
59 merged and the updates applied to the releng Jenkins production silo.
61 Please note that the combination of the Vagrant slave snapshot and the Jenkins
62 Spinup script is what defines a given slave. That means for instance that a
63 slave that is defined using the releng/builder/vagrant/basic-java-node Vagrant
64 and a Jenkins Spinup script of releng/builder/jenkins-script/controller.sh
65 (as the dynamic_controller slave is) is the full definition of the realized
66 slave. Jenkins starts a slave using the snapshot created that has been saved
67 from when the vagrant was last run and once the instance is online it then
68 checks out the releng/builder repo and executes two scripts. The first is the
69 basic_settings.sh which is a baseline for all of the slaves and the second is
70 the specialization script that does any syste updates, new software installs
71 or extra environment tweaks that don't make sense in a snapshot. After all of
72 these scripts have executed Jenkins will finally attach the slave as an actual
73 slave and start handling jobs on it.
75 ### Pool: Rackspace - Docker
77 <table class="table table-bordered">
79 <td><b>Jenkins Label</b><br/> dynamic_docker</td>
80 <td><b>Slave Template name</b><br/> rk-f20-docker</td>
81 <td><b>Vagrant Definition</b><br/> releng/builder/vagrant/ovsdb-docker</td>
82 <td><b>Spinup Script</b><br/> releng/builder/jenkins-scripts/docker.sh</td>
86 A Fedora 20 system that is configured with OpenJDK 1.7 (aka Java7) and
87 docker. This system was originally custom built for the test needs of
88 the OVSDB project but other projects have expressed interest in using
94 ### Pool: Rackspace DFW
96 <table class="table table-bordered">
98 <td><b>Jenkins Label</b><br/> dynamic_verify</td>
99 <td><b>Slave Template name</b><br/> rk-c-el65-build</td>
100 <td><b>Vagrant Definition</b><br/> releng/builder/vagrant/basic-builder</td>
101 <td><b>Spinup Script</b><br/> releng/builder/jenkins-scripts/builder.sh</td>
105 A CentOS 6 build slave. This system has OpenJDK 1.7 (Java7) and OpenJDK
106 1.8 (Java8) installed on it along with all the other components and
107 libraries needed for building any current OpenDaylight project. This is
108 the label that is used for all basic -verify and -daily- builds for
114 <table class="table table-bordered">
116 <td><b>Jenkins Label</b><br/> dynamic_merge</td>
117 <td><b>Slave Template name</b><br/> rk-c-el65-build</td>
118 <td><b>Vagrant Definition</b><br/> releng/builder/vagrant/basic-builder</td>
119 <td><b>Spinup Script</b><br/> releng/builder/jenkins-scripts/builder.sh</td>
123 See dynamic_verify (same image on the back side). This is the label that
124 is used for all basic -merge and -integration- builds for projects.
129 ### Pool: Rackspace DFW - Devstack
131 <table class="table table-bordered">
133 <td><b>Jenkins Label</b><br/> dynamic_devstack</td>
134 <td><b>Slave Template name</b><br/> rk-c7-devstack</td>
135 <td><b>Vagrant Definition</b><br/> releng/builder/vagrant/ovsdb-devstack</td>
136 <td><b>Spinup Script</b><br/> releng/builder/jenkins-scripts/devstack.sh</td>
140 A CentOS 7 system purpose built for doing OpenStack testing using
141 DevStack. This slave is primarily targeted at the needs of the OVSDB
142 project. It has OpenJDK 1.7 (aka Java7) and other basic DevStack related
148 ### Pool: Rackspace DFW - Integration
150 <table class="table table-bordered">
152 <td><b>Jenkins Label</b><br/> dynamic_robot</td>
153 <td><b>Slave Template name</b><br/> rk-c-el6-robot</td>
154 <td><b>Vagrant Definition</b><br/> releng/builder/vagrant/integration-robotframework</td>
155 <td><b>Spinup Script</b><br/> releng/builder/jenkins-scripts/robot.sh</td>
159 A CentOS 6 slave that is configured with OpenJDK 1.7 (Java7) and all the
160 current packages used by the integration project for doing robot driven
161 jobs. If you are executing robot framework jobs then your job should be
162 using this as the slave that you are tied to. This image does not
163 contain the needed libraries for building components of OpenDaylight,
164 only for executing robot tests.
169 ### Pool: Rackspace DFW - Integration Dynamic Lab
171 <table class="table table-bordered">
173 <td><b>Jenkins Label</b><br/> dynamic_controller</td>
174 <td><b>Slave Template name</b><br/> rk-c-el6-java</td>
175 <td><b>Vagrant Definition</b><br/> releng/builder/vagrant/basic-java-node</td>
176 <td><b>Spinup Script</b><br/> releng/builder/jenkins-scripts/controller.sh</td>
180 A CentOS 6 slave that has the basic OpenJDK 1.7 (Java7) installed and is
181 capable of running the controller, not building.
186 <table class="table table-bordered">
188 <td><b>Jenkins Label</b><br/> dynamic_java</td>
189 <td><b>Slave Template name</b><br/> rk-c-el6-java</td>
190 <td><b>Vagrant Definition</b><br/> releng/builder/vagrant/basic-java-node</td>
191 <td><b>Spinup Script</b><br/> releng/builder/jenkins-scripts/controller.sh</td>
195 See dynamic_controller as it is currently the same image.
200 <table class="table table-bordered">
202 <td><b>Jenkins Label</b><br/> dynamic_mininet</td>
203 <td><b>Slave Template name</b><br/> rk-c-el6-mininet</td>
204 <td><b>Vagrant Definition</b><br/> releng/builder/vagrant/basic-mininet-node</td>
205 <td><b>Spinup Script</b><br/> releng/builder/jenkins-scripts/mininet.sh</td>
209 A CentOS 6 image that has mininet, openvswitch v2.0.x, netopeer and
210 PostgreSQL 9.3 installed. This system is targeted at playing the role of
211 a mininet system for integration tests. Netopeer is installed as it is
212 needed for various tests by Integration. PostgreSQL 9.3 is installed as
213 the system is also capable of being used as a VTN project controller and
214 VTN requires PostgreSQL 9.3.
219 <table class="table table-bordered">
221 <td><b>Jenkins Label</b><br/> dynamic_mininet_fedora</td>
222 <td><b>Slave Template name</b><br/> rk-f21-mininet</td>
223 <td><b>Vagrant Definition</b><br/> releng/builder/vagrant/basic-mininet-fedora-node</td>
224 <td><b>Spinup Script</b><br/> releng/builder/jenkins-scripts/mininet-fedora.sh</td>
228 Basic Fedora 21 system with ovs v2.3.x and mininet 2.2.1
233 ### Pool: Rackspace DFW - Matrix
235 <table class="table table-bordered">
237 <td><b>Jenkins Label</b><br/> matrix_master</td>
238 <td><b>Slave Template name</b><br/> rk-c-el6-matrix</td>
239 <td><b>Vagrant Definition</b><br/> releng/builder/vagrant/basic-java-node</td>
240 <td><b>Spinup Script</b><br/> releng/builder/jenkins-scripts/matrix.sh</td>
244 This is a very minimal system that is designed to spin up with 2 build
245 instances on it. The purpose is to have a location that is not the
246 Jenkins master itself for jobs that are executing matrix operations
247 since they need a director location. This image should not be used for
248 anything but tying matrix jobs before the matrx defined label ties.
253 # Creating Jenkins Jobs
255 Jenkins Job Builder takes simple descriptions of Jenkins jobs in YAML format, and uses them to configure Jenkins.
257 * [Jenkins Job Builder](http://ci.openstack.org/jenkins-job-builder/) \(JJB\)
260 OpenDaylight releng/builder gerrit project
262 * [releng/builder](https://git.opendaylight.org/gerrit/#/admin/projects/releng/builder)
265 ## Jenkins Job Builder Installation
268 [Docker](https://www.docker.com/whatisdocker/) is an open platform used to
269 create virtualized Linux containers for shipping self-contained applications.
270 Docker leverages LinuX Containers \(LXC\) running on the same operating system
271 as the host machine, whereas a traditional VM runs an operating system over
274 docker pull zxiiro/jjb-docker
275 docker run --rm -v ${PWD}:/jjb jjb-docker
277 The Dockerfile that created that image is
278 [here](https://github.com/zxiiro/jjb-docker/blob/master/Dockerfile).
279 By default it will run:
283 Using the volume mount "-v" parameter you need to mount a directory containing
284 your YAML files as well as a configured jenkins.ini file if you wish to upload
285 your jobs to the Sandbox.
289 Jenkins Jobs in the releng silo use Jenkins Job Builder so if you need to test
290 your Jenkins job against the Sandbox you will need to install JJB.
292 The templates below depend on a modified JJB version to add support for some
293 missing features needed by our Jenkins instance. You can download JJB from
296 git clone https://git.openstack.org/openstack-infra/jenkins-job-builder
298 Before installing JJB make sure following python modules are installed (see
305 * python-jenkins>=0.4.1
308 Follow steps in README.rst to install JJB:
310 sudo python setup.py install
312 Notes for Mac: [instructions here](https://github.com/openstack-infra/jenkins-job-builder).
313 The <tt>sudo python setup.py install</tt> seems to work better than the
314 version using brew and pip.
316 Note: Some Linux distributions already contain a JJB package, usually with
317 version too low to work correctly with Releng templates. You may need to
318 uninstall the corresponding Linux package (or find another workaround) before
319 proceeding with steps from *README.rst*.
321 Update: Here is a link to e-mail with suggestions on how to install and
322 upgrade JJB properly:
323 https://lists.opendaylight.org/pipermail/integration-dev/2015-April/003016.html
325 ## Jenkins Job Templates
327 The ODL Releng project provides 4 job templates which can be used to
330 ### Verify Job Template
334 The Verify job template creates a Gerrit Trigger job that will trigger when a
335 new patch is submitted to Gerrit.
337 Verify jobs can be retriggered in Gerrit by leaving a comment that says
340 ### Merge Job Template
344 The Merge job template is similar to the Verify Job Template except it will
345 trigger once a Gerrit patch is merged into the repo. It also automatically
346 runs the Maven goals **source:jar** and **javadoc:jar**.
348 This job will upload artifacts to https://nexus.opendaylight.org on
351 Merge jobs can be retriggered in Gerrit by leaving a comment that says
354 ### Daily Job Template
356 The Daily (or Nightly) Job Template creates a job which will run on a build on
357 a Daily basis as a sanity check to ensure the build is still working day to
360 ### Sonar Job Template
362 Trigger: **run-sonar**
364 This job runs Sonar analysis and reports the results to
365 [OpenDaylight's Sonar dashboard(https://sonar.opendaylight.org).
367 **Note:** Running the "run-sonar" trigger will cause Jenkins to remove it's
368 existing vote if it's already -1 or +1'd a comment. You will need to re-run
369 your verify job (recheck) after running this to get Jenkins to put back the
372 The Sonar Job Template creates a job which will run against the master branch,
373 or if BRANCHES are specified in the CFG file it will create a job for the
374 **First** branch listed.
376 ### Integration Job Template
378 The Integration Job Template create a job which runs when a project that your
379 project depends on is successfully built. This job type is basically the same
380 as a verify job except that it triggers from other jenkins jobs instead of via
381 Gerrit review update. The dependencies are listed in your project.cfg file
382 under the **DEPENDENCIES** variable.
384 If no dependencies are listed then this job type is disabled by default.
388 Trigger: **test-integration**
390 This job runs a full integration test suite against your patch and reports
391 back the results to Gerrit. This job is maintained by the integration project
392 and you just need to leave a comment with trigger keyword above to activate it
393 for a particular patch.
395 **Note:** Running the "test-integration" trigger will cause Jenkins to remove
396 it's existing vote if it's already -1 or +1'd a comment. You will need to
397 re-run your verify job (recheck) after running this to get Jenkins to put back
400 Some considerations when using this job:
402 * The patch test verification takes some time (~ 2 hours) + consumes a lot of
403 resources so it is not meant to be used for every patch
404 * The system test for master patches will fail most of the times because both
405 code and test are unstable during the release cycle (should be good by the
407 * Because of the above, patch test results has to be interpreted most of the
408 times by a system test knowable person, the integration group can help with
411 ## Basic Job Configuration
413 To create jobs based on the above templates you can use the example
414 template which will create 6 jobs (verify, merge, and daily jobs for both
415 master and stable/helium branch).
417 Run the following steps from the repo (i.e. releng/builder) root to create
418 initial job config. This script will produce a file in
419 jjb/\<project\>/\<project\>.yaml containing your project's base template.
421 python scripts/jjb-init-project.py <project-name>
424 python scripts/jjb-init-project.py aaa
426 # Note: The optional options below require you to remove the 1st line
427 # comment in the produced template file otherwise the auto
428 # update script will overwrite the customization next time it
429 # is run. See Auto Update Job Templates section below for more
432 # Optionally pass the following options:
434 # -s / --streams : List of release streams you want to create jobs for. The
435 # first in the list will be used for the Sonar job.
436 # (defaults to "beryllium")
437 # -p / --pom : Path to pom.xml to use in Maven build (defaults to pom.xml)
438 # -g / --mvn-goals : With your job's Maven Goals necessary to build
439 # (defaults to "clean install")
440 # Example : -g "clean install"
442 # -o / --mvn-opts : With your job's Maven Options necessary to build
443 # (defaults to empty)
444 # Example : -o "-Xmx1024m"
446 # -d / --dependencies : A comma-seperated (no spaces) list of projects
447 # your project depends on.
448 # This is used to create an integration job that
449 # will trigger when a dependent project-merge job
450 # is built successfully.
451 # Example : aaa,controller,yangtools
453 # -t / --templates : Job templates to use
454 # (defaults: verify,merge,daily,integration,sonar)
456 # Example : verify,merge,daily,integration
458 If all your project requires is the basic verify, merge, and
459 daily jobs then using the job template should be all you need to
460 configure for your jobs.
462 ### Auto Update Job Templates
464 The first line of the job YAML file produced by the script will contain
465 the words # REMOVE THIS LINE IF... leaving this line will allow the
466 releng/builder autoupdate script to maintain this file for your project
467 should the base template ever change. It is a good idea to leave this
468 line if you do not plan to create any complex jobs outside of the
471 However if your project needs more control over your jobs or if you have
472 any additional configuration outside of the standard configuration
473 provided by the template then this line should be removed.
475 #### Tuning templates
477 Additionally the auto-updater does allow some small tweaks to the template
478 so that you can take advantage of the template while at the same time
479 tuning small aspects of your jobs. To take advantage of this simply create
480 a file in your project's jjb directory called **project.cfg** with the
481 following contents and tune as necessary. If there is a parameter you do
482 NOT want to tune simply remove the parameter or comment out the line with a
485 JOB_TEMPLATES: verify,merge,sonar
488 jdks: openjdk7,openjdk8
492 MVN_GOALS: clean install javadoc:aggregate -DrepoBuild -Dmaven.repo.local=$WORKSPACE/.m2repo -Dorg.ops4j.pax.url.mvn.localRepository=$WORKSPACE/.m2repo
493 MVN_OPTS: -Xmx1024m -XX:MaxPermSize=256m
494 DEPENDENCIES: aaa,controller,yangtools
495 ARCHIVE_ARTIFACTS: *.logs, *.patches
497 Note: BRANCHES is a list of branches you want JJB to generate jobs for, the
498 first branch will be the branch that reports Sonar analysis. Each branch must
499 additionally define a "jdks:" section listing the jdks the verify jobs should
500 run tests against for the branch; additionally the first jdk listed will be
501 used as the default jdk for non-verify type jobs.
505 It is also possible to take advantage of both the auto updater and creating
506 your own jobs. To do this, create a YAML file in your project's sub-directory
507 with any name other than \<project\>.yaml. The auto-update script will only
508 search for files with the name \<project\>.yaml. The normal \<project\>.yaml
509 file can then be left in tact with the "# REMOVE THIS LINE IF..." comment so
510 it will be automatically updated.
513 [https://jenkins.opendaylight.org/sandbox](https://jenkins.opendaylight.org/sandbox/)
515 The Sandbox instance's purpose is to allow projects to test their JJB setups
516 before merging their code over to the Releng Master silo. It is configured
517 similarly to the Master instance above however it cannot publish or vote in
520 If your project requires access to the Sandbox please open a Help Desk ticket
521 and provide us with your ODL ID.
523 #### Notes regarding the Sandbox
525 * Jobs automatically deleted Saturday @ 08:00 UTC (12:00 AM PST / 01:00 AM PDT)
526 * Committers can login and configure Jenkins jobs directly here (unlike on the
528 * Configuration mirrors the master silo when possible
529 * Can NOT upload artifacts to Nexus
530 * Can NOT vote on Gerrit
532 #### Using the Sandbox
534 Before starting using the sandbox make sure you have Jenkins Job Builder
535 properly installed in your setup. Refer Jenkins Job Builder Installation
536 section of this guide.
538 If you do not already have access, open a helpdesk ticket to request access to
539 the sandbox instance (Integration committers will have access by default).
541 1. Clone a copy of the releng/builder repo from https://git.opendaylight.org/gerrit/#/admin/projects/releng/builder
542 2. cp jenkins.ini.example jenkins.ini
543 3. Edit the jenkins.ini file at the root of the repo
544 * Set your ODL username and password (make sure to uncomment the lines)
545 * Set the URL to https://jenkins.opendaylight.org/sandbox
547 It is good practice to test that your JJB files are valid before pushing using
548 the test command. If you see no Exceptions or Failures after running the
549 following command your templates should be good for pushing.
551 The last parameter is the name of the job you want to push to Jenkins so if
552 your job template name is **{project}-csit-3node-periodic-{functionality}-{install}-{stream}**
553 you will need to expand manually the variables {project}, {functionality},
554 {install}, and {stream} to the exact job you want created in the Sandbox for
555 example **openflowplugin-csit-1node-periodic-longevity-only-beryllium**. Please
556 do not push ALL jobs to the Sandbox and only jobs you actually intend to test.
558 **Note:** the below command examples are being executed from the root of the
559 builder repo, and assume the "jenkins.ini" file is located there.
561 jenkins-jobs --conf jenkins.ini test jjb/ <job-name>
562 jenkins-jobs --conf jenkins.ini test jjb/ openflowplugin-csit-periodic-1node-cds-longevity-only-master
564 Expect to see an XML file describing the build job in </maven2-moduleset> tags
565 on STOUT. If you dont see any XML check that you have assigned values to the
566 parameters between {} in the YAML files. For example {project}
568 Once this is complete you can push your JJB jobs to the sandbox with the
571 jenkins-jobs --conf jenkins.ini update jjb/ <job-name>
572 jenkins-jobs --conf jenkins.ini update jjb/ openflowplugin-csit-periodic-1node-cds-longevity-only-beryllium
574 **Important Note:** When pushing with jenkins-jobs command it will print out a
575 message similar to the one below to inform you how many jobs JJB is pushing
576 online. If the number is greater than 1 (or the number of jobs you passed to
577 the command to push) then you are pushing too many jobs and should **ctrl+c**
578 to cancel the upload.
580 INFO:jenkins_jobs.builder:Number of jobs generated: 1
585 docker run --rm -v ${PWD}:/jjb zxiiro/jjb-docker
587 # To upload jobs to the sandbox
588 # Please ensure that you include a configured jenkins.ini in your volume mount
589 docker run --rm -v ${PWD}:/jjb zxiiro/jjb-docker jenkins-jobs --conf jenkins.ini update . openflowplugin-csit-periodic-1node-cds-longevity-only-master