Merge "Capture logs for all packaging jobs"

Merge "Bug 8625: Remove Nitrogen Karaf 3 jobs"

Call openstack directly

Rather than using $PYTHON $OPENSTACK as have acceptable virtualenv paths
now.

Also fix ShellCheck in this file.

Change-Id: Ib2cda5d1b4c5f90138d92479f5ed786182fd247a
Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>

Fix virtualenv usage in integration

Virtualenvs are in /tmp/v/{name}.
This inclused "openstack" venv.
Add missing --upgrade to every pip install line.

Change-Id: Ifd0dc429b6e088e9d86dea9ee57d33b2363ba7eb
Signed-off-by: Vratko Polak <vrpolak@cisco.com>

Bug 8625: Remove Nitrogen Karaf 3 jobs

All jobs related to Karaf 3 in Nitrogen should be removed,
as TSC voted on not spending any cycles on Karaf 3 in Nitrogen.

Karaf 4 is the new default for CSIT in Nitrogen,
so it is expected some jobs start failing
for projects not fully working on Karaf 4.

Change-Id: Iae504c2acf6576f15cffa7f91b4cdcb37599ddbe
Signed-off-by: Vratko Polak <vrpolak@cisco.com>

Capture logs for all packaging jobs

Using publishers is recommended for jobs to refer logs
at anytime, and also would help minimizing the loads on Jenkins.

Change-Id: Ie0d766394693a44ba58d1f01d73d2dd6e1d26224
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>

Merge "Install virtualenvs into /tmp/v"

Merge "Install py3 virtualenv only on Ubuntu 16.04"

Merge "Remove bash validate for provision scripts"

Remove bash validate for provision scripts

With shellcheck in place we dont need to validate scripts with
bash, which is outdated.

Change-Id: I6e98c6e58afeea350c89c88ed6c6508e6170a8ad
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>

Install virtualenvs into /tmp/v

The kernel has a limit on how long a shebang can be and causes build
issues when we store virtualenvs in $WORKSPACE for jobs that have very
long names. Move virtualenv installs into /tmp/v will solve this issue
for good. Also no need for workarounds like `$PYTHON -m pip` anymore.

Change-Id: Iaab7eeb5144ba26ba8ea58efc631c4d7a5c0f60c
Jira: RELENG-156
Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>

Install py3 virtualenv only on Ubuntu 16.04

python3-virtualenv package is not available on Ubuntu 14.04
which causes the packer job to fail

Change-Id: Ia99b372c6123abd282e4a37bfb4b1f1ef0582613
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>

Update to global-jjb v0.3.4

Change-Id: Id9998913889c9cda39a79cf364012ec6d1297691
Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>

Bump global-jjb to v0.3.3

Change-Id: Iedb0721fd53a849d407428d9dd6ec0fcc31da5e0
Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>

Rename odl-global-settings to global-settings

Simplify the naming to be consistent with other LF projects for this
file.

Change-Id: I875f9a5c51074627b7c617957a185e3e2fb5ba80
Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>

Merge "Remove sysstats change to avoid duplicity"

Add odlparent release job

- Bump global-jjb to v0.3.2 to fix typo
- Fix lf-infra jobs to use updated tox template names
- Setup odlparent release job for Nitrogen.

Change-Id: Ia726352870b85983c2373e903004887d8079621b
Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>

Merge "Make Boron autorelease run weekly"

Update to global-jjb v0.3.0

Change-Id: I011d09c09980f2728d207e616bfffd3772532d66
Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>

Make Boron autorelease run weekly

Change-Id: I157a31c8a0dff89329a19be1e449d422071664ac
Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>

Merge "Include python development for Ubuntu images"

Remove sysstats change to avoid duplicity

Now we have sysstats as part of lf-infra-publish which is integrated
into global-jjb repo, therefore remove all sysstat changes.

Change-Id: Idc046f18726c2afafb6578d1adcfc83dfc2614fb
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>

Merge "Minor improvements to release notes job"

Include python development for Ubuntu images

Building netifaces extension requires headers from
python-dev, without which pip install fails.

Move all python deps into baseline for Ubuntu and
CentOS images

Jira: releng-182
Change-Id: I9828d3c0aa855651e0daeb63b8451c9e041a3177
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>

Merge "Change array in loop bashism to satisfy shellcheck"

Change array in loop bashism to satisfy shellcheck

Warning: I'm not totally sure this is right, but it seems to be the
recommended way to keep the same behavior we had but be explicit that
we want it. If shellcheck was actually catching a bug and we don't want
the old behavior, we should do the loop recommendation.

https://github.com/koalaman/shellcheck/wiki/SC2199

Change-Id: Ibcb2e88ad4df7d1789747b137e2135154ca8435a
Signed-off-by: Daniel Farrell <dfarrell@redhat.com>

Merge "Update global-jjb to latest master"

Minor improvements to release notes job

Tweak the job so that it is more compact and uses only the necessary
declarations.

- Move generate-release-notes job declaration into autorelease projects
  declaration as it uses the same variables as the rest of the jobs.
- Switch to a 4g VM as generating release notes does not require a large
  system.
- Only keep logs for 7 days as opposed to 30 days.
- Choosing strategy should be default as this is not a job that is
  triggered by Gerrit.
- Set timeout to 10 minutes as the job appears to run in less then 5
  minutes.
- Remove unnecessary parameters from Maven call as the mvn command does
  not deploy anything to Nexus.
- Use global-jjb macros.

Change-Id: I71d8b18656cf941e0ce691493011ffb10e289480
Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>

Restart Spectrometer weekly

The Spectrometer frontend tends to crash so lets give it a kick once a
week.

Change-Id: I0a730a87960ddb8fed591d8c1dc245ca31614cc1
Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>

Merge "Update CentOS7 devstack images"

Merge "Update CentOS7 java-builder images"

Update global-jjb to latest master

To pull in the python -m change.

Change-Id: I5236239b51f2a821452ebbd5948cc6611228aaff
Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>

Update cloud image list docs

Change-Id: I5500c6a52a9f96edbcd3bf40df428255aa4c4da8
Signed-off-by: jenkins-releng <jenkins-releng@opendaylight.org>

Update CentOS7 devstack images

Change-Id: Ia81158d323514866dcbbb0e08e51d12d84116349
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>

Update CentOS7 java-builder images

Change-Id: Iddba4b59111a17394846d4dc3deed74e275145e6
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>

Merge "Del. lispflowmapping-distribution3-check-nitrogen"

Merge "Remove 6wind/quagga build temporarily"

Enable sysstat on Ubuntu and CentOS images

Change-Id: I99b18d1de3f06995532d46193e06a0b679ce3222
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>

Merge "Fix shellcheck SC1090 cannot follow source"

Fix shellcheck SC1090 cannot follow source

Change-Id: Idf0bce1211eca6529ffe1f27335ac895c91180ea
Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>

Remove infrautils-validate-autorelease job

These jobs only work if infrautils are part of autorelease which it
currently isn't.

Change-Id: Id4a8283efe7231897f86bced857191f8d0596e04
Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>

Merge "Update cloud image list docs"

Merge "Install git-review using pip install in virtialenv"

Update cloud image list docs

Change-Id: I9dacf89359f6a7750abb6a1b6ea7ed4279480e0c
Signed-off-by: jenkins-releng <jenkins-releng@opendaylight.org>

Merge "Remove dependencies by projects no longer active"

Merge "Bake fedora-packager into java-builder image"

Remove 6wind/quagga build temporarily

No response from the zrpcd maintaners to update the scripts.
Once the issue is resovled we can enabled this on images.

https://lists.opendaylight.org/pipermail/dev/2017-May/003724.html
https://lists.opendaylight.org/pipermail/dev/2017-May/003698.html

Change-Id: I0d05e45528d549a3ba32542e946b23198724fecc
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>

Merge "Remove baked packages from spinup script"

Remove dependencies by projects no longer active

These projects are no longer active ODL projects so remove their
dependencies from our packer builds.

Change-Id: I30c7a6ceba3ca7fb2bd628713faa81ef44ceb513
Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>

Bake fedora-packager into java-builder image

Make the image boot a little more quickly by baking it into
java-builder packer build.

Change-Id: Ia002c6327a676021198199ea054ff3f62abf2553
Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>

Remove baked packages from spinup script

These packages are built as part of the packer build process so no need
to additionally install them during system spinup.

Change-Id: I8223279cd057124c66b3449ad92857ceb282eea1
Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>

Update to latest java-builder

Change-Id: I7f784c47d38efd28e9949eac052e27d3590e3396
Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>

Merge "Revert "Increase packer job timeout to 120 minutes""

Add verify jobs for global-jjb and lftools

Update global-jjb to latest master to pull in tox-verify job and add
verify jobs for lf-infra releng projects.

Change-Id: I47fa9d2edfde867da440eefcc39558bab34d2c7e
Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>

Fix shellcheck issue with implicent array concat

Change-Id: I6b16514a9eb5745d64041eb165a89705cb0d8f0d
Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>

Revert "Increase packer job timeout to 120 minutes"

Since we reverted https://git.opendaylight.org/gerrit/58436
90 minutes is more than enough to build our images.

This reverts commit 94fe23608b05f41be25f41ac15ed1e9e4b76ebe7.

Change-Id: I8a655c2da2e7caa4ed24147be23bdf6bc60d47bf
Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>

Merge "Revert "Make packer jobs build sequentially""

Revert "Make packer jobs build sequentially"

This reverts commit 514da68ea959e926bafc996521a434af8fc9147a.

Change-Id: I1e1a62a7053350713b95e04cd7243c4c000ef7fd
Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>

Increase packer job timeout to 120 minutes

90 minutes wasn't enough to build java-builder in the public cloud when
the public cloud is running slow. Increase to 2 hrs.

Change-Id: Idbaa1a1c5225c94a9f3870d2f3965c3c79deb700
Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>

Merge "Increase packer job timeout to 90 minutes"

Increase packer job timeout to 90 minutes

Packer takes long to build so split it into it's own project definition
so we can set reasonable build timeouts without increasing the other
builder jobs which only need 10 minutes to run.

Change-Id: I73efe2c583f4661b81233ac448e203788857822b
Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>

No need to pin Cabal for Ubuntu

Ubuntu comes with a sufficiently new version of Cabal to install
ShellCheck so no need to downgrade it.

Change-Id: I7c13adc1cc1982cde6a72b98a845086de803aed4
Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>

Update cloud image list docs

Change-Id: I8e2db643c88a6d7f54a0320db76fc51c4436f24d
Signed-off-by: jenkins-releng <jenkins-releng@opendaylight.org>

Del. lispflowmapping-distribution3-check-nitrogen

The lispflowmapping project wishes to disable its Karaf 3 features and
local distributions in the build, to be able to re-join autorelease as
soon as possible. This patch removes the
lispflowmapping-distribution3-check-nitrogen job which fails as a
result.

Change-Id: Id6e9df99ccef2c2e78f966cb58ed28148be29900
Signed-off-by: Lorand Jakab <lojakab@cisco.com>

Build and install shellcheck from the Haskell repo

Pull shellcheck from Haskell repo so that we can pull in a newer version
than is available in CentOS 7 and Ubuntu Trusty repos.

Change-Id: I710c05dab4a088647426190a00d9cf7ab9b175b0
Co-Authored-By: Thanh Ha <thanh.ha@linuxfoundation.org>
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>

Merge "Update cloud image list docs"

Merge "Add validate-autorelease job back for infrautils"

Merge "Update Pin RS centos7 image to working version"

Merge "DO NOT fail autorelease if file scripts fail"

Update Pin RS centos7 image to working version

This is an update to commmit 7bcb9c78fbfe76d90527d269bc1721d233800676
where other templates which use CentOS7 are changed.

Jira: releng-118
Change-Id: Ia99264894a95a423a9d15499e9928b3fa731f20c
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>

Install git-review using pip install in virtialenv

The version of git-review supports --reviewers option which is not
available with the version of the package on centos7 repositories.

Updated the change to handle long workspace names with pip
modules instead of pip, inspired by global-jjb change #5093

Change-Id: I598350d6778760ea320a1128db46da0f80c711db
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>

Merge "Update packer job to build devstack ocata image"

Add validate-autorelease job back for infrautils

Change-Id: Ic08aef29f19b8ac88eb3b0fa86d5bcd50d45ab61
Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>

DO NOT fail autorelease if file scripts fail

There are 2 scripts which create a file in int/dist which are sure to
fail if int/dist is not included in autorelease. We noticed this because
of Nitrogen work.

Change-Id: I7a7123c551b420f838b281c810dc1711ef5c90d2
Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>

Update global-jjb to latest master

Change-Id: Ia25e85040eaaf06c2944d85d6d74ca8f44efbe28
Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>

Update cloud image list docs

Change-Id: I539db0aed37b9cc4c3bcfb16114bf612dd6187c0
Signed-off-by: jenkins-releng <jenkins-releng@opendaylight.org>

Update packer job to build devstack ocata image

Change-Id: Ifaba56767fb4c23988fbb21327b86feed65441f1
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>

Add devstack-ocata packer template

Change-Id: Ic63c14179ac13a59989ba892466eba5f232849d4
Signed-off-by: Jamo Luhrsen <jluhrsen@redhat.com>

Also install git-review on Ubuntu systems

Update to d036ff0f which forgot to also include Ubuntu systems.

Change-Id: I93babd64194c7640fbc5a05fb84699eabda0476d
Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>

Create a wrapper function for apt-get install

Make the loop reusable.

Change-Id: I1213d07acf19586c2f2775acce488acc39a282d2
Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>

Merge "Update Pin RS centos7 image to working version"

Update Pin RS centos7 image to working version

This is an update to commmit 7bcb9c78fbfe76d90527d269bc1721d233800676
where other templates which use java-builder are changed.

Jira: RELENG-118
Change-Id: Ida6bd041674b4767db5c2030288ef1b036958239
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>

Merge "Archive the docs html directory"

Remove validate-ar jobs for non-nitrogen projects

Per this thread:
https://lists.opendaylight.org/pipermail/tsc/2017-June/007428.html

Remove all projects except for aaa, controller, md-sal, netconf,
odlparent, and yangtools.

Change-Id: I88a748e47c36dfc543c33fbaf563e72531e5fa37
Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>

Pin RS Public cloud images to known working

We're having issues with RS Public Cloud when building newer Cent OS
images. This patch pins the public cloud base image we build against to
the last known working one.

Jira: RELENG-118
Change-Id: Iad0ec2d3e312c22042c15973cab0eef58c61fe9c
Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>

Merge "Update default RPM snap build version to Nitrogen"

Archive the docs html directory

This patch using the new lf-infra-publisher ensures that docs only
archives the html directory rather than including the structure leading
to the directory.

Change-Id: Ie2ff63690203ea7023f2578d4f52719f9d386a52
Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>

Remove opendaylight-shiplogs macro

We've migrated to using lf-infra-publish macro from global-jjb.

Change-Id: I5ec61c188d90bc9d220ace2296bcc57c1544cf2f
Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>

Merge "Migrate project jobs to use lf-infra-publish"

Merge "Migrate int/test jobs to use lf-infra-publish"

Merge "Migrate int/dist to use lf-infra-publish"

Merge "Migrate docs jobs to use lf-infra-publish"

Merge "Migrate autorelease jobs to lf-infra-publish"

Merge "Migrate releng job templates to lf-infra-publish"

Use -m pip from python instead of pip directly

When the workspace is too long, as can happen with long
job names, there is trouble running commands in the
virtualenv.

this is trying to find all the places this would be
needed, but specifically to address a few current
problems like [0] which sometimes completes but cannot
push logs because the job name is too long.

The idea came from shague [1] which came from
openstack [2].

[0] https://jenkins.opendaylight.org/releng/user/jluhrsen/my-views/view/netvirt%20csit/job/netvirt-csit-1node-openstack-newton-nodl-v2-upstream-stateful-snat-conntrack-carbon/
[1] https://git.opendaylight.org/gerrit/#/c/51699
[2] https://review.openstack.org/#/c/399754

Change-Id: I9e99893d8de69ef0eb7b36309983cedc0befb549
Signed-off-by: Jamo Luhrsen <jluhrsen@redhat.com>

Remove auto job creation for Beryllium

now that Carbon is released, we are no longer dealing
with Beryllium

Change-Id: Iae74afd9742b03b9ba1e6907a8e3bfd335c4a00e
Signed-off-by: Jamo Luhrsen <jluhrsen@redhat.com>

Auto Update CSIT Jobs to run

Change-Id: I469ed201c7fccc20a93e39d9a8accdc4941e5bad
Signed-off-by: jenkins-releng <jenkins-releng@opendaylight.org>
Signed-off-by: Jamo Luhrsen <jluhrsen@redhat.com>

Update default RPM snap build version to Nitrogen

Change-Id: I6b2a19faf31b4c103ce1f4b3956703d68fdff504
Signed-off-by: Daniel Farrell <dfarrell@redhat.com>

Add a workaround for missing taglist.log

Since we fail the build now if any line in a script fails this is
causing validate-autorelease to fail unexpectedly. Temporarily put in
place a workaround on the problematic line so that it passes until we
put in a proper fix.

Change-Id: I31999f0f78d5ca360f20fa14536310924263c0a4
Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>

Merge "Add Boron branch and csit jobs for jsonrpc"