1 Logic for building OpenDaylight's upstream Source RPMs.
5 OpenDaylight's Source RPMs (SRPMs) are built using the logic provided here.
7 When new builds are defined, the new SRPMs are uploaded to the CentOS
8 Community Build System's Koji-based RPM build system. From there they are
9 built into ready-to-install noarch RPMs and hosted for consumption.
11 Data that differs per-build is is defined in the `build_vars.yaml` YAML
12 configuration file. The build logic consumes that dynamic data, uses
13 JinJa2 templates to generate RPM spec files and builds the RPMs/SRPMs they
16 See the Templates vs Macros section for details about why we use this design.
18 ## Vagrant Build Environment
20 The included Vagrantfile defines a consistent, known-working and easily
21 shared environment. It supports both VM and container-based providers.
25 Current machine states:
27 default not created (virtualbox)
30 [vagrant@localhost vagrant]$ ls /vagrant/
31 build.py build_vars.yaml cache connect.sh Vagrantfile <snip>
36 The Vagrantfile defines a Docker provider, enabling easy access to build.py
37 in a container. The general command format is:
40 $ vagrant docker-run -- <flags to build.py>
43 To pass 5.0 (Boron) as the version to build:
46 $ vagrant docker-run -- -v 5 0
49 Dockerfile can be also used directly to build container image:
52 $ docker build -t "odl_rpm" .
53 $ docker run -v $(pwd):/build odl_rpm -v 5 0
58 The dynamic aspects of a build, such as ODL and RPM version info, have all
59 been extracted to a single YAML configuration file. For most RPM updates,
60 only that configuration file should need to be updated by humans.
62 The variables available for configuration are documented below. A build
63 definition should define all supported variables.
65 ### RPM Build Variables
69 The OpenDaylight major (element) version number of the release to build.
71 For example, Hydrogen is 1.x.x, Helium is 2.x.x, Lithium is 3.x.x and
72 so on down the periodic table.
76 The OpenDaylight minor (SR) version number of the release to build.
78 OpenDaylight provides periodic Service Releases (SRs) for each currently
79 supported Major release. See the [Hydrogen][12], [Lithium][13] or
80 [Beryllium][13] release plan schedules for more information.
84 The OpenDaylight patch version of the release to build (unused pending CR).
86 This version number isn't currently used by OpenDaylight, as there aren't
87 releases more frequent than Service Releases. However, the upstream
88 OpenDaylight community is working towards supporting a Continuous Release (CR)
89 model. Once that exists, this patch version will denote CR-level releases.
93 RPM version for the given ODL major.minor.patch version.
95 In addition to OpenDaylight's version, RPMs themselves have versions. These
96 are called "release versions". For a given OpenDaylight major.minor.patch
97 version, there will be one or more major.minor.patch-rpm_release RPMs.
101 Version of ODL systemd unitfile to download and package in ODL RPM.
103 The version of OpenDaylight's systemd unitfile in the `packaging/rpm/unitfiles`
104 directory specified by this git commit hash is downloaded from the [Int/Pack
105 repo][16] and consumed by OpenDaylight's RPM builds as an RPM spec file Source.
109 Elemental codename for the ODL release, including SR if applicable.
111 Examples include "Helium-SR4", "Lithium" and "Lithium-SR1".
115 The ODL RPM repackages the tarball build artifact produced by ODL's
116 autorelease build, because building Java projects from source directly
117 to RPMs is very difficult with current tooling. This is the URL to the
118 tarball ODL build to repackage into an RPM.
122 Entry in the RPM .spec file's changelog for this RPM.
124 When the RPM spec file template is specialized into per-build static RPM spec
125 files, a changelog is generated using these entries.
127 The changelog entry must follow a specific format. It's best to follow the
128 examples provided by existing build definitions closely. The `rpmbuild` tool
129 will fail and complain fairly clearly if the format isn't correct.
131 ### Testing Build Logic
133 Some `build.py` logic is covered by Python unit tests.
138 $ python -m unittest test_build
141 ## Building SRPMs/RPMs
143 The `build.py` helper script is used for building OpenDaylight SRPMs/RPMs.
145 All SRPM builds are done in the included Vagrant environment. SRPMs are then
146 uploaded to the CentOS Community Build System, built into noarch RPMs there
147 and hosted for distribution.
149 The `build.py` helper script builds noarch RPMs in addition to the (actually
150 used) SRPMs. The noarch RPMs are meant for quickly sanity checking that a
151 build worked as expected. They can be installed in the Vagrant build box or
152 another sandbox, then inspected.
154 The `build.py` helper script can build a set of SRPMs/RPMs based on provided
158 [vagrant@localhost ~]$ /vagrant/build.py -h
159 usage: build.py [-h] [-v [major minor patch rpm [major minor patch rpm ...]]]
162 -h, --help show this help message and exit
163 -v [major minor patch rpm [major minor patch rpm ...]], --version [major minor patch rpm [major minor patch rpm ...]]
164 RPM version(s) to build
167 The `-v`/`--version` flag accepts a version number. Any build that matches
168 the portions provided will be built. If more than one build matches the
169 portions provided, all matching builds will be executed.
171 For example, `build.py -v 3` would execute the builds that match the regex
172 `3.*.*-*`. OpenDaylight 3.0.0-1 and 3.1.0-1, for example.
174 To only build a single RPM definition, provide enough version info to make
175 the match unique. For example, `build.py -v 2 4 0 1` could only match one
176 definition (Helium SR4, 2.4.0-1).
178 The `build.py` script uses the `cache/cache.py` script to handled downloading
179 and caching the artifacts required for the requested builds. Artifacts are
180 cached in the `packaging/rpm/cache/` directory.
182 The `build.py` script also uses the `specs/build_specs.py` script to generate
183 static RPM spec files from the `packaging/rpm/specs/opendaylight.spec` JinJa2
184 templates and the dynamic build data provided in `build_vars.yaml`. The
185 resulting RPM spec files are passed to `rpmbuild` by `build.py` to build
186 specific ODL SRPMs/RPMs.
188 ## Working with the ODL RPM
190 The familiar RPM-related commands apply to the OpenDaylight RPM.
192 ### Installing OpenDaylight via a local RPM
194 To install a local OpenDaylight RPM, perhaps as a sanity check after a
195 build, use `sudo yum install -y <path to ODL RPM>`.
197 Here's a walk-through of an install and the resulting system changes.
200 # Note that there's nothing in /opt before the install
201 [vagrant@localhost ~]$ ls /opt/
202 # Note that there are no ODL systemd files before the install
203 [vagrant@localhost ~]$ ls /usr/lib/systemd/system | grep -i opendaylight
204 # Install an ODL RPM. Yum will handle installing ODL's Java dependency.
205 [vagrant@localhost ~]$ sudo yum install -y /vagrant/cache/<RPM filename>
206 # Note that ODL is now installed in /opt
207 [vagrant@localhost ~]$ ls /opt/
209 # Note that there's now a systemd .service file for ODL
210 [vagrant@localhost ~]$ ls /usr/lib/systemd/system | grep -i opendaylight
214 ### Uninstalling OpenDaylight's RPM
216 To uninstall OpenDaylight's RPM, use `sudo yum remove -y opendaylight`.
218 Here's a walk-through of the uninstall and the resulting system changes.
221 # Note that ODL is installed in /opt/
222 [vagrant@localhost vagrant]$ ls /opt/
224 # Note that there's a systemd .service file for ODL
225 [vagrant@localhost vagrant]$ ls /usr/lib/systemd/system | grep -i opendaylight
227 # Uninstall the ODL RPM
228 [vagrant@localhost vagrant]$ sudo yum remove -y opendaylight
229 # Note that ODL has been removed from /opt/
230 [vagrant@localhost vagrant]$ ls /opt/
231 # Note that the ODL systemd .service file has been removed
232 [vagrant@localhost vagrant]$ ls /usr/lib/systemd/system | grep -i opendaylight
235 ## Managing OpenDaylight via systemd
237 The OpenDaylight RPM ships with systemd support.
239 ### Starting OpenDaylight via systemd
242 [vagrant@localhost vagrant]$ sudo systemctl start opendaylight
243 [vagrant@localhost vagrant]$ sudo systemctl status opendaylight
244 ● opendaylight.service - OpenDaylight SDN Controller
245 Loaded: loaded (/usr/lib/systemd/system/opendaylight.service; disabled)
246 Active: active (running) since Tue 2015-07-14 21:09:30 UTC; 4s ago
247 Docs: https://wiki.opendaylight.org/view/Main_Page
248 http://www.opendaylight.org/
249 Process: 18216 ExecStart=/opt/opendaylight/bin/start (code=exited, status=0/SUCCESS)
250 Main PID: 18223 (java)
251 CGroup: /system.slice/opendaylight.service
252 └─18223 /usr/bin/java -Djava.security.properties=/opt/opendaylight/etc/odl.jav...
254 Jul 14 21:09:30 localhost.localdomain systemd[1]: Started OpenDaylight SDN Controller.
257 ### Stopping OpenDaylight via systemd
260 [vagrant@localhost vagrant]$ sudo systemctl stop opendaylight
261 [vagrant@localhost vagrant]$ sudo systemctl status opendaylight
262 opendaylight.service - OpenDaylight SDN Controller
263 Loaded: loaded (/usr/lib/systemd/system/opendaylight.service; disabled)
264 Active: inactive (dead)
265 Docs: https://wiki.opendaylight.org/view/Main_Page
266 http://www.opendaylight.org/
270 ## Connecting to the Karaf shell
272 A few seconds after OpenDaylight is started, its Karaf shell will be accessible.
274 The `connect.sh` script is provided as an example of how to connect to the Karaf shell.
277 # Assuming you've started ODL
278 [vagrant@localhost ~]$ /vagrant/connect.sh
279 Installing sshpass. It's used connecting non-interactively
281 opendaylight-user@root>
284 Additionally, here's an example of connecting manually (password: `karaf`):
287 [vagrant@localhost vagrant]$ ssh -p 8101 -o StrictHostKeyChecking=no karaf@localhost
288 Authenticated with partial success.
289 Password authentication
292 ________ ________ .__ .__ .__ __
293 \_____ \ ______ ____ ____ \______ \ _____ ___.__.| | |__| ____ | |___/ |_
294 / | \\____ \_/ __ \ / \ | | \\__ \< | || | | |/ ___\| | \ __\
295 / | \ |_> > ___/| | \| ` \/ __ \\___ || |_| / /_/ > Y \ |
296 \_______ / __/ \___ >___| /_______ (____ / ____||____/__\___ /|___| /__|
297 \/|__| \/ \/ \/ \/\/ /_____/ \/
300 Hit '<tab>' for a list of available commands
301 and '[cmd] --help' for help on a specific command.
302 Hit '<ctrl-d>' or type 'system:shutdown' or 'logout' to shutdown OpenDaylight.
304 opendaylight-user@root>^D
305 Connection to localhost closed.
306 [vagrant@localhost vagrant]$
309 ## CentOS Community Build System
311 After building SRPMs as described above, we use the [CentOS Community Build
312 System][1] to build and host noarch RPMs for official consumption.
316 OpenDaylight's CBS usage is under the umbrella of the CentOS Network Function
317 Virtulalization Special Interest Group (NFV SIG).
319 Committers to OpenDaylight Integration/Packaging [should][2] have CentOS CBS
320 commit access, which actually manifests as Koji "build" permissions.
322 To get access, [file a bug][3] against CentOS with `Project:Buildsys` and
323 `Category:community buildsys`. Include the Koji username you'd like, your
324 GPG public key (inclusion in OpenDaylight's web of trust is strongly
325 encouraged), your email address and the name of the SIG ("NFV" in our case).
326 For an example, see [Daniel Farrell's request][4]. For additional info, see
327 the [Quickstart section of the CBS docs][5].
329 As noted in those docs, you'll need to get the SIG chair to leave a note
330 on your bug to verify that you should have access. Dave Neary is the
331 current NFV SIG chair. The [Int/Pack][15] PTL (currently [Daniel Farrell][6])
332 can help you get his cooperation.
336 The CentOS CBS is Koji-based, so the main way to interface with it via the
337 Koji CLI client. There's also a [Koji web UI][1].
339 Other than the [main CBS wiki][7] and Koji's help output, there aren't
340 substantial Koji docs. These docs strive to cover everything that's
341 commonly needed for ODL's RPMs.
343 Once you get access to the CBS, you'll be emailed a tarball with the
344 configuration and certificates required by the Koji CLI when interacting
345 with the CBS's Koji server.
347 Koji expects the contents of that tarball in the `~/.koji` directory.
351 # Actually symlinks that point to files I decrypt only when necessary
352 clientca.crt client.crt config serverca.crt
355 Verify that you have the appropriate permissions.
358 $ koji list-permissions
363 If you're building a new OpenDaylight major version, you'll need to raise
364 a bug like [this one][8] to get the target created. Once it's created,
365 a set build tags will be created for that major version. For now, we're
366 only using the `*-candidate` tags.
369 $ koji list-tags | grep "opendaylight-.-candidate"
370 nfv7-opendaylight-2-candidate
371 nfv7-opendaylight-3-candidate
372 nfv7-opendaylight-4-candidate
375 For new tags, the `opendaylight` package needs to be added before a build.
378 $ koji add-pkg --owner "dfarrell07" nfv7-opendaylight-3-candidate opendaylight
381 When submitting a build, it's typically a good idea to verify that everything
382 is in working order using the `--scratch` option.
385 $ koji build --scratch nfv7-opendaylight-3-el7 opendaylight-3.0.0-1.el7.src.rpm
388 If everything works, submit the actual build.
391 $ koji build nfv7-opendaylight-3-el7 opendaylight-3.0.0-1.el7.src.rpm
394 You can monitor the build using the Koji CLI `watch-logs` and `watch-task`
395 commands, or the [Koji web UI's][1] output of the same information.
397 If the RPM build is successful, it will appear [on the CBS][9] under the
398 appropriate `nfv7-opendaylight-<major version>-testing` repo. The resulting
399 RPM can be installed directly or via a [package manager config file][10].
402 $ sudo curl -o /etc/yum.repos.d/opendaylight-3-candidate.repo \
404 $ sudo yum install -y opendaylight
407 ## Templates vs Macros
409 This section documents *why* we use the YAML+Py+JinJa2 design we do, whereas
410 the sections above document *how*.
412 Typically, RPM builds use the macro mechanism provided by RPM spec files
413 and the `rpmbuild` tool to customize specs for specific builds. This works
414 well for default macros that are gathered from the system automatically, like
415 `disttag` and `unitdir`.
417 However, once we needed to support multiple RPM builds at the same time, we
418 needed to either accept a large amount of logic duplication or use custom
419 macros to specialize a single spec file per-build. Custom macros work fine
420 for building locally, as they can be passed to `rpmbuild` as params from a
421 script like `build.sh` (as shown by past versions of this codebase). However,
422 once an SPRM is created and the spec file is packaged into it, it needs to
423 be uploaded to the CentOS Community Build System's Koji-based build system
424 to be built into a noarch RPM and hosted for distribution. The only way to
425 define custom macros in CBS builds is to create a second RPM to install a
426 macro definition file in the correct place on the build box. Maintaining a
427 second RPM is an unacceptable increase in complexity, and it's unclear if
428 Koji handles macro RPMs correctly anyway (potentially buggy, at least very
431 Given this inability to define dynamic, custom macros in Koji-based builds,
432 but the need to extract variables to avoid manual duplication of a large
433 amount of spec file logic, a design that uses dynamic per-build data +
434 templates to build simi-static (only default macros) RPM spec files emerged.
435 The specifics (YAML, Python, JinJa2) were chosen because of their use in
436 other tools used by [Int/Pack][15], like Packer, Ansible and Puppet.
438 [1]: http://cbs.centos.org/koji/
440 [2]: https://trello.com/c/cgQmevT8/209-additional-access-to-centos-cbs
442 [3]: https://bugs.centos.org/
444 [4]: https://bugs.centos.org/view.php?id=8879
446 [5]: https://wiki.centos.org/HowTos/CommunityBuildSystem#head-00dad77f5720f3a984b0fac9f9bacac52047f73f
448 [6]: https://wiki.opendaylight.org/view/User:Dfarrell07
450 [7]: https://wiki.centos.org/HowTos/CommunityBuildSystem
452 [8]: https://bugs.centos.org/view.php?id=9472
454 [9]: http://cbs.centos.org/repos/
456 [10]: https://github.com/opendaylight/integration-packaging/tree/master/rpm/example_repo_configs
458 [11]: https://bugs.centos.org/view.php?id=9098#c23768
460 [12]: https://wiki.opendaylight.org/view/Simultaneous_Release:Helium_Release_Plan#Schedule
462 [13]: https://wiki.opendaylight.org/view/Simultaneous_Release:Lithium_Release_Plan#Schedule
464 [14]: https://wiki.opendaylight.org/view/Simultaneous_Release:Beryllium_Release_Plan#Schedule
466 [15]: https://wiki.opendaylight.org/view/Integration/Packaging
468 [16]: https://git.opendaylight.org/gerrit/gitweb?p=integration/packaging.git;a=tree;h=refs/heads/master;hb=refs/heads/master