the `RelEng/Autorelease`_ project, and primarily takes the form of
`Autorelease's Jenkins jobs`_.
-Autorelease builds every project from source. There are jobs for each current
-OpenDaylight release, as well as the version under development.
-
+Autorelease builds every project from source. Artifact versions are rewritten
+from the -SNAPSHOT suffixes in version control to release versions, like
+-Carbon-SR1 or -Nitrogen. This contrasts with distribution jobs, which build
+only a few projects from source and use -SNAPSHOT artifact versions. This makes
+autorelesae builds slow, but identical to actual releases, whereas distribution
+builds are fast but slightly less similar to official releases.
Daily Releases
--------------
`Autorelease's Jenkins jobs`_ run daily for every active branch, including
master.
-- `Beryllium autorelease job`_
- `Boron autorelease job`_
- `Carbon autorelease job`_
+- `Nitrogen autorelease job`_
Each of those jobs, when the build is successful, produces build artifacts that
include an OpenDaylight distribution. To download the distribution, pick an
and search for "staging repository with ID". Find the repositoiry ID, which
will be of the form "autorelease-1432". Navigate to `OpenDaylight's Nexus`_ and
find the staging repository with the same name. Drill down into the directory
-tree org/opendaylight/integration/distribution-karaf/ to find the `build
-artifacts`_. Autorelease build artifacts are persevered for 60 days.
+tree org/opendaylight/integration/distribution-karaf/ to find the build
+artifacts. Autorelease build artifacts are persevered for 60 days.
Autorelease jobs trigger OpenDaylight's distribution tests when they complete.
To see the test results, go to integration-distribution-test-<branch> job's
Jenkins page, find the job that started after the autorelease in question
-finished. Open it and explore `Subprojects` section for test results of all
-the jobs triggered. For example, in case of Boron, you can find the list and
+finished. Open it and explore the subprojects section for test results of all
+the jobs triggered. For example, in case of Nitrogen, you can find the list and
the results of jobs triggered `here`_.
-The OpenDaylight Integration/Test project recently audited all tests to remove
-false negatives that were cluttering this report with failures that didn't
-imply a problem with OpenDaylight, but with test logic. If there are failures,
-especially new ones, pay attention to which OpenDaylight features they affect.
-If you don't load that Karaf feature, it shouldn't be relevant to you.
-
The latest successful autorelease builds can also be easily found by using the
`staging/org/opendaylight/integration/distribution-karaf/`_
-and look for 0.4.5-Beryllium-SR5, 0.5.3-Boron-SR3 or 0.6.0-Carbon or similar
+and look for 0.5.4-Boron-SR4 or 0.6.1-Carbon-SR1, 0.7.0-Nitrogen or similar
staging repositories. However, the artifacts in these repositories are not
static - they are replaced each time new artifacts are generated. Use the
"autorelease-XXXX" repositories described above for semi-persistent URLs.
the Release Engineering and Integration/Test teams, and if all's well blesses
the build as an official release. The build's Nexus staging repo is then
promoted to a release repo and publicized (example: `opendaylight.release/org
-/opendaylight/integration/distribution-karaf/0.5.2-Boron-SR2`_). Official
+/opendaylight/integration/distribution-karaf/0.6.0-Carbon`_). Official
releases are persevered forever.
For more information about OpenDaylight releases, including timelines, see the
.. _RelEng/Autorelease: https://git.opendaylight.org/gerrit/gitweb?p=releng/autorelease.git;a=tree;h=refs/heads/master;hb=refs/heads/master
.. _Autorelease's Jenkins jobs: https://jenkins.opendaylight.org/releng/view/autorelease/
-.. _Beryllium autorelease job: https://jenkins.opendaylight.org/releng/view/autorelease/job/autorelease-release-beryllium/
.. _Boron autorelease job: https://jenkins.opendaylight.org/releng/view/autorelease/job/autorelease-release-boron/
.. _Carbon autorelease job: https://jenkins.opendaylight.org/releng/view/autorelease/job/autorelease-release-carbon/
+.. _Nitrogen autorelease job: https://jenkins.opendaylight.org/releng/view/autorelease/job/autorelease-release-nitrogen/
.. _OpenDaylight's Nexus: https://nexus.opendaylight.org/content/repositories/
-.. _build artifacts: https://nexus.opendaylight.org/content/repositories/autorelease-1432/org/opendaylight/integration/distribution-karaf/0.5.0-Boron-RC1/
-.. _here: https://jenkins.opendaylight.org/releng/job/integration-distribution-test-boron/
+.. _here: https://jenkins.opendaylight.org/releng/job/integration-distribution-test-nitrogen/
.. _staging/org/opendaylight/integration/distribution-karaf/: https://nexus.opendaylight.org/content/repositories/staging/org/opendaylight/integration/distribution-karaf/
-.. _opendaylight.release/org /opendaylight/integration/distribution-karaf/0.5.2-Boron-SR2: https://nexus.opendaylight.org/content/repositories/opendaylight.release/org/opendaylight/integration/distribution-karaf/0.5.2-Boron-SR2/
+.. _opendaylight.release/org /opendaylight/integration/distribution-karaf/0.6.0-Carbon: https://nexus.opendaylight.org/content/repositories/opendaylight.release/org/opendaylight/integration/distribution-karaf/0.6.0-Carbon/
.. _Release Plans: https://wiki.opendaylight.org/view/Release_Plan
.. _build.py: https://github.com/opendaylight/integration-packaging/blob/master/deb/build.py
.. _here: https://github.com/opendaylight/integration-packaging/blob/master/deb/README.markdown
.. _build_deb job: https://jenkins.opendaylight.org/releng/job/packaging-build-deb-master/
-.. _given build description: https://jenkins.opendaylight.org/releng/job/packaging-build-deb-master/build?delay=0sec
+.. _given build description: https://jenkins.opendaylight.org/releng/job/packaging-build-deb-master/
=======================
Unlike autorelease builds, which build every project from source, distribution
-jobs only build a few Karaf features. This makes them much quicker (minutes
-instead of ~11 hours), and therefore suitable for CI testing. End-users like
-OPNFV should stick with autorelease jobs.
+jobs only build a few Karaf features. The other artifacts are pulled pre-built
+from OpenDaylight's Nexus repository and packaged into the Karaf distribution.
+This makes them much quicker (minutes instead of ~4 hours).
+The other major difference between autorelease and distribution job builds is
+that distribution jobs use the -SNAPSHOT artifact version suffixes that are
+actually stored in version control, whereas autorelease builds rewrite versions
+to use the suffix for the next release, like -Carbon-SR1 or -Nitrogen. Because
+of this, distribution builds are sometimes called "snapshot builds".
-Snapshot Builds
----------------
+For each active branch, builds created by distribution jobs can be found in the
+subdirectories at `opendaylight.snapshot/org/opendaylight/integration
+/distribution-karaf/`_. Each build artifact is versioned with a timestamp and
+unique, incrementing build number.
+
+Distribution Builds Triggered by Merge Jobs
+-------------------------------------------
Distribution job builds are typically kicked off when a patch is merged into
a project. Projects define `<project>`-merge-<branch> Jenkins jobs, which are
-kicked off by a Gerrit merge events. To find the merge job for a Gerrit, look
+kicked off by Gerrit merge event. To find the merge job for a Gerrit, look
for comments from the jenkins-releng user like "Build Started
https://jenkins.opendaylight.org/releng/job/netvirt-merge-boron/216/".
Alternatively, browse a project's Jenkins tab and look at the recent runs.
-For example, go to https://jenkins.opendaylight.org/releng/, select Merge-Boron
-and you'll find the list of all projects merge jobs in the format
-`<project>`-merge-boron. Click any to view the recent build job details and
+For example, go to https://jenkins.opendaylight.org/releng/, select
+Merge-Carbon and you'll find the list of all project merge jobs in the format
+`<project>`-merge-carbon. Click any to view the recent build job details and
logs.
-For each active branch, snapshot builds created by distribution jobs can be
-found in the subdirs at `opendaylight.snapshot/org/opendaylight/integration
-/distribution-karaf/`_. The maven-metadata.xml tells about the different
-versions and the latest one. The different subdirectories (say, 0.5.2-SNAPSHOT)
-contain all the corresponding version (0.5.2 here) builds stored by time (time
-included in the artifact name).
+Custom Distributions
+--------------------
-Recent build artifacts info for a given branch can be found in the XML's
-`<content-item>` at `opendaylight.snapshot/content/org/opendaylight/integration
-/distribution-karaf/`_.
+Distributions can be built with an additional set of unmerged patches. The
+`integration-multipatch-test-<branch>`_ jobs allow users to specify a set of
+patches to cherry-pick onto a project's source code before building. This is
+very useful for testing complex changes that impact multiple projects.
+To build a custom distribution that includes a set of unmerged patches, first
+make sure you have permission to trigger Jenkins jobs. Send an email to the
+OpenDaylight Helpdesk (helpdesk@opendaylight.org) to request access. Be sure
+to include your Linux Foundation user ID in the request.
-Custom Distributions
---------------------
+Once you can trigger Jenkins jobs, navigate to the Jenkins web UI for the
+multipatch-test job of the branch you're interested in. Make sure you're
+logged in, then click on the "Build with Parameters" link in the sidebar.
+The only parameter that requires configuration is PATCHES_TO_BUILD. This is
+a CSV list of patches in project[=checkout][:cherry-pick]* format. For each
+given project, the job will checkout 0 or 1 specified patches, then cherry-pick
+0 or more additional patches on top of that checkout. If no checkout is
+specified, cherry-picks will be done on top of the tip of the branch of the
+multipatch-test job you're using.
+
+For example, to build with a single unmerged patch from NetVirt:
+
+ netvirt:59/50259/47
+
+Because of the colon, this would cherry-pick the change on top of the tip
+of the multipatch-test job branch.
+
+To build with the same NetVirt patch, but by directly checking it out, use
+an equals sign.
+
+ netvirt=59/50259/47
+
+This will be the same thing if the patch has recently been rebased on top
+of the tip of the branch, but may be different if the patch is based on a
+different set of patches.
+
+To build with checked-out patches from Genius and NetVirt:
+
+ genius=32/53632/9,netvirt=59/50259/47
+
+To checkout a patch from controller, then cherry-pick another on top of it:
+
+ controller=61/29761/5:45/29645/6
-Distributions can be built with an additional set of unmerged patches. `See
-this wiki`_.
+The numbers in the changeset are the Gerrit change ID of the patch (middle
+number) and the patchset of the Gerrit (last number). The first number is
+just the last two digits of the Gerrit change ID (I'm not sure why this is
+necessary). I belive it's required that patches be listed in the order the
+projects are built (NetVirt depends on Genius, so Genius is listed first).
+For the definitive explination of how the multipatch job works, see the `JJB
+source that defines it`_.
.. _opendaylight.snapshot/org/opendaylight/integration /distribution-karaf/: https://nexus.opendaylight.org/content/repositories/opendaylight.snapshot/org/opendaylight/integration/distribution-karaf/
-.. _opendaylight.snapshot/content/org/opendaylight/integration /distribution-karaf/: https://nexus.opendaylight.org/service/local/repositories/opendaylight.snapshot/content/org/opendaylight/integration/distribution-karaf/
-.. _See this wiki: https://wiki.opendaylight.org/view/Integration/Test/Running_System_Tests#Running_System_Tests_Using_Custom_Distribution_Built_From_Multiple_Patches
+.. _integration-multipatch-test-<branch>: https://jenkins.opendaylight.org/releng/search/?q=integration-multipatch-test
+.. _JJB source that defines it: https://github.com/opendaylight/releng-builder/blob/master/jjb/integration/include-raw-integration-multipatch-distribution-test.sh
.. noqa
.. toctree::
- :maxdepth: 2
+ :maxdepth: 5
+ packages
autorelease-builds
distribution-job-builds
- packages
Packages
========
-Some builds are packaged as RPMs or .debs for easy consumption by downstream
-users.
+Builds can be packaged as RPMs or .debs. To provide inputs into OpenDaylight's
+Continious Delivery pipelines to downstream projects, many builds are
+automatically packaged. Every succesful autorelease build is packaged as an
+RPM. Every day, the latest distribution snapshot build is packaged as an RPM.
+This keeps new artifacts flowing even if some projects are breaking
+autorelease. Custom packages can be built from custom distributions, for
+example with yet-to-be merged patches that need system testing.
.. noqa
.. toctree::
RPMs
====
+OpenDaylight has a mature RPM Continuous Delivery pipeline. Every autorelease
+build is automatically packaged as an RPM, and even if autorelease is broken
+a daily job builds the latest distribution snapshot build into an RPM.
-Automated RPM Builds
---------------------
-
-OpenDaylight Integration/Packaging has added support for many variations of
-fully automated RPM builds.
-
-
-Build RPM Job
-^^^^^^^^^^^^^
-
-The Jenkins `build_rpm job`_ builds an ODL RPM described by the `given Jenkins
-build parameters`_, using the `build.py`_ script.
+RPMs can be passed to test jobs that install them, start OpenDaylight with it's
+systemd service, connect to the Karaf shell and verify basic functionality.
+RPMs are hosted on the CentOS Community Build system repositories. Some repos
+are updated very frequently with the latest builds, while others are permeate
+homes of official releases.
-Build Latest Snapshot Job
-^^^^^^^^^^^^^^^^^^^^^^^^^
+Developers can build custom RPMs with pre-merge patches for testing by first
+creating a custom distribution with the integration-multipatch-test job and
+then feeding the resulting artifact to the packaging-build-rpm job.
-For a given major version, you can build an RPM from the latest snapshot by
-passing `- -build-latest-snap` to build.py.
+Build Jobs
+----------
-The Jenkins `build_rpm_snap job`_ builds the latest snapshot into RPM. It
-extracts ODL version info from the artifact's URL to build artifacts. The
-necessary JJB params to pass are OpenDaylight's major and minor version to
-build, sysd_commit (version of ODL systemd unitfile to download and package
-in RPM), changelog name and email.
-
-
-CentOS CBS RPMs
----------------
-
-OpenDaylight's RPMs are built and hosted on the CentOS Community Build System's
-Koji build servers.
+OpenDaylight Integration/Packaging has added support for many variations of
+fully automated RPM builds.
-All OpenDaylight RPMs, and the SRPMs with tarballs from the builds described
-previously, are permanently available for download `here`_.
+packaging-build-rpm
+^^^^^^^^^^^^^^^^^^^
-See the `Deployment#RPM`_ wiki for more information about RPMs.
+The `packaging-build-rpm job`_ is the primary way to build an RPM from an
+OpenDaylight distribution (built by autorelease or the snapshot distribution
+job). It accepts a set of `parameters`_ that can be used to configure the build
+and passes them to the `RPM build logic in Integration/Packaging's repo`_. The
+resulting artifacts are hosted on Jenkins for up to a week. The job actually
+produces both a noarch RPM and source RPM. The noarch RPM can be passed to test
+jobs for validation. The source RPM can be downloaded to a system with the
+required credentials and then pushed to the CentOS Community Build system to
+be built into a noarch RPM on their servers and hosted in their repos.
+packaging-build-rpm-snap
+^^^^^^^^^^^^^^^^^^^^^^^^
-.. _build_rpm job: https://jenkins.opendaylight.org/releng/job/packaging-build-rpm-master/
-.. _given Jenkins build parameters: https://jenkins.opendaylight.org/releng/job/packaging-build-rpm-master/build?delay=0sec
-.. _build.py: https://github.com/opendaylight/integration-packaging/blob/master/rpm/build.py
-.. _build_rpm_snap job: https://jenkins.opendaylight.org/releng/job/packaging-build-rpm-snap-master/
-.. _here: http://cbs.centos.org/koji/packageinfo?packageID=755
-.. _Deployment#RPM: https://wiki.opendaylight.org/view/Deployment#RPM
+The `packaging-build-rpm-snap job`_ packages the most recent snapshot
+distribution build from a given branch as an RPM. This could be used by a
+developer to test code that was just merged, but which has not been included
+in an autorelease build yet. The job is also triggered daily, to ensure that
+OpenDaylight's Continuous Delivery pipeline is fed new builds even if
+autorelease is broken.
+
+Test Jobs
+---------
+
+packaging-test-rpm
+^^^^^^^^^^^^^^^^^^
+
+The `packaging-test-rpm job`_ accepts a link to an RPM and validates it. It
+installs the package with the system's package manager, starts OpenDaylight's
+systemd service, verifies that it's reported as active, connects to the Karaf
+shell and checks that some key bundles are present.
+
+Repositories
+------------
+
+CentOS
+^^^^^^
+
+While most RPM builds are triggered automatically in OpenDaylight's Jenkins,
+some RPMs are promoted to be hosted in OpenDaylight's CentOS repositories.
+There are a series of repos that are updated at varying frequencies, from
+testing repos that are updated with pre-release versions very frequently to
+release repos that are the permeate home of official OpenDaylight releases.
+
+Testing Repositories
+....................
+
+Repositories with the -testing suffix are updated very frequently with
+pre-release versions of OpenDaylight from the appropriate branch. New RPMs
+replace the old ones, so installing from these repos will always provide the
+most recent versions.
+
+Testing repos for Boron, Carbon and Nitrogen:
+
+- `nfv7-opendaylight-5-testing`_
+- `nfv7-opendaylight-6-testing`_
+- `nfv7-opendaylight-7-testing`_
+
+Release Repositories
+....................
+
+Repositories with the -release suffix host official OpenDaylight releases. They
+are updated infrequently to never, and will host their release artifacts
+forever. Release repos are subdivided into two groups based version numbers.
+Repositories with both a major and minor version number (52, 53, 60) are pinned
+to a specific OpenDaylight release or service release (Boron SR2 5.2.0, Boron
+SR3 5.3.0, Carbon 6.0.0). Repositories with only a major version (5, 6) will
+always host the latest service release from that major release. If a new SR
+come out, the repo will get the update (Boron SR4 will replace Boron SR3).
+
+Release repos for the latest Boron and Carbon service releases:
+
+- `nfv7-opendaylight-5-release`_
+- `nfv7-opendaylight-6-release`_
+
+Release repos that will permanently host specific Boron and Carbon releases:
+
+- `nfv7-opendaylight-50-release`_
+- `nfv7-opendaylight-51-release`_
+- `nfv7-opendaylight-52-release`_
+- `nfv7-opendaylight-53-release`_
+- `nfv7-opendaylight-60-release`_
+
+Repository Configuration Files
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+While it's possible to install RPMs directly (`dnf install -y <URL>`), it's
+often easier to use a repository configuration file to install whatever the
+latest RPM is in a given repo.
+
+The OpenDaylight Integration/Packaging project provides `example repo config
+files for each official repository`_.
+
+Package managers like Yum and DNF will automatically find repo configuration
+files placed in the /etc/yum.repos.d/ directory. Curl them into place with
+something like:
+
+ sudo curl -o /etc/yum.repos.d/opendaylight-7-testing.repo \
+ "https://git.opendaylight.org/gerrit/gitweb?p=integration/packaging.git;a=blob_plain;f=rpm/example_repo_configs/opendaylight-7-testing.repo"
+
+Standard install commands will now find the repository as expected.
+
+ sudo dnf install -y opendaylight
+
+Custom RPMs
+-----------
+
+It's possible for developers to build custom RPMs, typically with unmerged
+patches that need system testing. First, use the integration-multipatch-test
+job to create a custom distribution that includes the set of unmerged patches.
+See the `Custom Distributions <distribution-job-builds.rst#Custom
+Distributions>`_ section for extensive docs. Once you have a custom
+distribution artifact, pass it to the `packaging-build-rpm job`_ to package it
+as an RPM. See the `packaging-build-rpm`_ section for docs.
+
+.. _packaging-build-rpm job: https://jenkins.opendaylight.org/releng/job/packaging-build-rpm-master/
+.. _parameters: https://jenkins.opendaylight.org/releng/job/packaging-build-rpm-master/build
+.. _RPM build logic in Integration/Packaging's repo: https://github.com/opendaylight/integration-packaging/blob/master/rpm/build.py
+.. _packaging-build-rpm-snap job: https://jenkins.opendaylight.org/releng/job/packaging-build-rpm-snap-master/
+.. _packaging-test-rpm job: https://jenkins.opendaylight.org/releng/job/packaging-test-rpm-master/
+.. _nfv7-opendaylight-5-testing: http://cbs.centos.org/repos/nfv7-opendaylight-5-testing/x86_64/os/Packages/
+.. _nfv7-opendaylight-6-testing: http://cbs.centos.org/repos/nfv7-opendaylight-6-testing/x86_64/os/Packages/
+.. _nfv7-opendaylight-7-testing: http://cbs.centos.org/repos/nfv7-opendaylight-7-testing/x86_64/os/Packages/
+.. _nfv7-opendaylight-5-release: http://cbs.centos.org/repos/nfv7-opendaylight-5-release/x86_64/os/Packages/
+.. _nfv7-opendaylight-6-release: http://cbs.centos.org/repos/nfv7-opendaylight-6-release/x86_64/os/Packages/
+.. _nfv7-opendaylight-50-release: http://cbs.centos.org/repos/nfv7-opendaylight-50-release/x86_64/os/Packages/
+.. _nfv7-opendaylight-51-release: http://cbs.centos.org/repos/nfv7-opendaylight-51-release/x86_64/os/Packages/
+.. _nfv7-opendaylight-52-release: http://cbs.centos.org/repos/nfv7-opendaylight-52-release/x86_64/os/Packages/
+.. _nfv7-opendaylight-53-release: http://cbs.centos.org/repos/nfv7-opendaylight-53-release/x86_64/os/Packages/
+.. _nfv7-opendaylight-60-release: http://cbs.centos.org/repos/nfv7-opendaylight-60-release/x86_64/os/Packages/
+.. _example repo config files for each official repository: https://git.opendaylight.org/gerrit/gitweb?p=integration/packaging.git;a=tree;f=rpm/example_repo_configs;hb=refs/heads/master
Code Review / integration / packaging.git / commitdiff