1 .. _transportpce-tox-guide:
9 `Tox` is a tool written in Python to ease tests automation and dependencies management.
10 It provides a command line tool that run tests inside a `Python virtual environment <https://docs.python.org/3/glossary.html#term-virtual-environment>`_.
12 This means that it will not modify your local system settings or package and executable files.
13 Instead, it uses a hidden folder (`.tox`) to install the required Python dependencies via pip
14 (`the package installer for Python <https://pip.pypa.io/>`_) before running tests.
16 You can find more details about tox at https://github.com/tox-dev/tox .
17 `Tox` is often used as a front-end to Continuous Integration servers.
19 For instance, `Linux Foundation globaljjb <https://globaljjbdocstest.readthedocs.io/en/latest/jjb/lf-python-jobs.html#tox-verify>`_
20 provides a gerrit-tox-verify profile.
21 This profile is `used by TransportPCE <https://git.opendaylight.org/gerrit/c/releng/builder/+/78656>`_
22 in OpenDaylight Jenkins CI.
24 `Tox` configuration and behavior is very portable across GNU+Linux distributions
25 and others UNIX-like systems or environments.
27 Once tox installed in a local environment with
28 for example the following command on Debian based systems::
30 $ sudo apt-get install tox
32 or on Red Hat based systems::
34 $ sudo yum install python-tox
36 the same test suite than the CI can be run locally by simply calling the tox command
37 from the project git clone local folder::
44 `Tox` configuration is written in the `tox.ini` file at the root folder of the Git project.
45 Please read `tox official documentation <https://tox.wiki/>`_ for more details.
46 For tox users, the most important parameter in the `[tox]` section is `envlist`.
47 It specifies which profiles to run by default (i.e. when tox is called without the option `-e`).
48 The option `-e` overrides this parameter and allows to choose which profiles to run.
53 will only run the `gitlint` profile.
56 $ tox -e gitlint,checkbashisms
58 will run the `gitlint` and `checkbashisms` profiles.
60 Profiles configuration are described in the `[testenv]` section.
61 Each profile specificities are usually configured in a subsection, for example the
62 subsection `[testenv:gitlint]` for the configuration of `Gitlint <https://jorisroovers.com/gitlint/>`_.
67 The `docs` profile itself is used to generate from the sources the HTML documentation of the project.
68 This docuementation can be found in `docs/_build/html/`
69 once the following command has been run locally::
73 The sources are written in the reStructuredText format a.k.a. RST and are located in the `docs/` folder.
75 `Sphinx` can also be used to check the validity of the URLs present in the sources
76 with the `docs-linkcheck` profile::
78 $ tox -e docs-linkcheck
80 False positives can be declared in the sphinx configuration file (usually `docs/conf.py`).
81 Both `docs` and `docs-linkcheck` profiles are run in the CI.
83 A third profile called `spelling` and based on `Sphinx` and `PyEnchant <https://pyenchant.github.io/pyenchant/>`_
84 can also be used as a spellchecker.
88 All docs profiles call the sphinx Python package.
89 Documentation and more details can be found at `sphinx home page <https://www.sphinx-doc.org/>`_ .
90 Web page templates are inherited from the Python dependency `lfdocs-conf` delared in `docs/requirements.txt`.
95 A few linter are also provided from tox profiles.
96 Some are even performed in the CI.
98 * `gitlint`. Check that the last commit message is well formatted.
99 * `pylint`. Lint Python tests scripts
100 * `autopep8`. Autoformat Python tests scripts according to PEP8 standard rules.
101 * `pyang`. Lint YANG files.
102 * `pyangformat`. Autoformat YANG files.
103 * `checkbashisms`. Detect bashisms in shell scripts.
109 `Pre-commit <https://pre-commit.com/>`_ is another wrapper for linters that relies on `Git Hooks <https://git-scm.com/docs/githooks>`_.
110 It is particularly useful to address common programming issues such as
111 triming trailing whitespaces or removing tabs.
113 `Pre-commit` configuration can be found in the `.pre-commit-config.yaml` file
114 at the Git project root.
115 `Pre-commit` hooks, like any other Git hooks, are run automatically when the
116 command `'git commit'` is called. This avoids forgetting running linters.
117 Although, `pre-commit` can also be called directly from its shell command.
118 This is what is currently performed in TransportPCE CI.
120 The `pre-commit` profile allows to call `pre-commit` inside tox virtualenv
121 without installing the `pre-commit` package in the local system,
122 what is pretty convenient::
126 This is also true to install/uninstall the corresponding Git hooks in your
127 Git folder thanks to the profiles `pre-commit-install`
128 and `pre-commit-uninstall`::
130 $ tox -e pre-commit-install
134 $ tox -e pre-commit-uninstall
136 Functional tests profiles
137 -------------------------
139 TransportPCE functional tests are Python scripts that allow to perform blackbox testing on a
141 They do not need tox to be performed locally and can be called directly from the shell or with
142 a launcher such as `nosetests <https://nose.readthedocs.io/>`_
143 (often available in Linux distributions packages under the name `python-nose`).
144 Currently, they require the presence of `Honeynode simulators <https://gitlab.com/Orange-OpenSource/lfn/odl/honeynode-simulator>`_
145 and the modification of the controller default OLM timers to speed-up the tests.
146 They are also supposed to be called within the tests folder::
150 These tests have been spread over several directories in `tranportpce_tests/`.
151 These directories are mostly named upon OpenROADM device versions.
152 Tests scripts files names are also numbered so that they are performed in a certain order.
153 To ease their integration in tox, a script `launch_tests.sh` can be used to call them.
154 For example, the following command::
156 $ ./launch_tests.sh pce
158 will call by default all the tests in the folder `tests/transportpce_tests/pce` with `nose`.
161 $ ./launch_tests.sh 1.2.1 portmapping
165 $ nosetests --with-xunit transportpce_tests/1.2.1/test01_portmapping.py
167 Several tests can be listed in the arguments. For example::
169 $ ./launch_tests.sh 1.2.1 portmapping topology
173 $ nosetests --with-xunit transportpce_tests/1.2.1/test01_portmapping.py
174 $ nosetests --with-xunit transportpce_tests/1.2.1/test03_topology.py
176 Also, some shell environment variables can be used to modify their default behavior.
177 For example the commands::
179 $ export LAUNCHER="python3"
180 $ ./launch_tests.sh 2.2.1
184 $ python3 transportpce_tests/2.2.1/test01_portmapping.py
188 $ export LAUNCHER="nosetests"
189 $ export USE_LIGHTY="True"
190 $ ./launch_tests.sh 7.1
194 $ nosetests transportpce_tests/7.1/test01_portmapping.py
196 but will ask tests script to use the controller `lighty.io <https://lighty.io/>`_
197 build instead of Karaf.
199 These variables are also understood inside tox virtualenv thanks to the `passenv` parameter
200 configured in `tox.ini`.
202 Tox TransportPCE functional tests support is split into several tox profiles.
203 Strictly spoken, only the following profiles perform functional tests as described above:
205 * `testsPCE`. To evaluate the Path Computation behavior.
206 * `tests121`. To evaluate the support of OpenROADM devices version 1.2.1 .
207 * `tests221`. To evaluate the support of OpenROADM devices version 2.2.1 .
208 * `tests71`. To evaluate the support of OpenROADM devices version 7.1 .
209 * `tests_hybrid`. To evaluate the controller behavior in a mixed environment with several versions of OpenROADM devices.
210 * `gnpy`. To evaluate the controller behavior when used in conjunction with `GNPy <https://github.com/Telecominfraproject/oopt-gnpy>`_. Requires `docker <https://www.docker.com/>`_.
211 * `nbinotifications`. To evaluate the controller north-bound interface notifications support. Requires `docker <https://www.docker.com/>`_.
213 Each of these profiles depend on the `buildcontroller` profile, which is simply
214 there to build the controller from sources and adapt OLM default timers.
215 They can also depend on `sims121` or `sims221` or `sims71` profiles to download
216 simulators of OpenROADM devices when needed.
217 Other profiles named from the pattern `build_karaf_testsXXX` have also been
218 added to configure separate karaf instances with alternate listening ports
219 in order to use concurrency.
221 The `depend` parameter in `tox.ini` allows tox to establish the most efficient
222 tests order strategy when calling tox without the `-e` option.
223 This is particularly important when the parallelized mode is enabled.
224 If tox is called locally with the option `-e`, profiles not specified to this
225 option but listed in the `depends` parameters are simply ignored.
226 This means you have to specify manually the `buildcontroller` or `simsXXX`
227 profiles if the controller was not build yet or the sims were not downloaded.
230 $ tox -e buildcontroller,sims121,tests121
232 or with karaf alternate builds::
234 $ tox -e buildcontroller,build_karaf_tests121,sims121,tests121
236 will build the controller and download simulators before running every functional
237 tests for OpenROADM devices 1.2.1.
238 Once that done, you only need to list the others sims versions profiles before
239 lauching hybrid tests::
241 $ tox -e sims221,sims71,tests_hybrid
243 or with karaf alternate builds::
245 $ tox -e build_karaf_tests_hybrid,sims221,sims71,tests_hybrid
247 Also the same way arguments can be passed to the `launch_tests.sh` script,
248 tests names can be passed as argument when calling the corresponding tox profiles.
252 $ tox -e tests121 portmapping
254 will launch by default the following command inside tox virtual environment::
256 $ nosetests --with-xunit transportpce_tests/1.2.1/test01_portmapping.py
260 $ tox -e tests121 "portmapping topology"
264 $ nosetests --with-xunit transportpce_tests/1.2.1/test01_portmapping.py
265 $ nosetests --with-xunit transportpce_tests/1.2.1/test03_topology.py
267 Note the necessity to use quotes here when listing several test names.
268 If you need to test the portmapping behavior for every OpenROADM devices versions::
270 $ tox -e tests121,tests221,tests71 portmapping
274 $ nosetests --with-xunit transportpce_tests/1.2.1/test01_portmapping.py
275 $ nosetests --with-xunit transportpce_tests/2.2.1/test01_portmapping.py
276 $ nosetests --with-xunit transportpce_tests/7.1/test01_portmapping.py
278 Idem for OLM with only OpenROADM devices versions 1.2.1 and 2.2.1 ::
280 $ tox -e tests121,tests221 olm
284 $ nosetests --with-xunit transportpce_tests/1.2.1/test05_olm.py
285 $ nosetests --with-xunit transportpce_tests/2.2.1/test08_olm.py
287 Profiles parrallelization
288 -------------------------
290 Tox profiles execution can be parallelized.
291 CI behavior can be configured from the `releng/builder` repository.
292 This is `the current configuration <https://git.opendaylight.org/gerrit/c/releng/builder/+/96557>`_
295 Locally, tox jobs are not parallelized by default.
296 You have to use the `-p` option to specify the level of concurrency::
308 The default parameter "auto" is based on the number of CPU cores,
309 which is a bad idea for TransportPCE functional tests.
310 Their most critical ressource is RAM, mostly because of the need
311 to launch several simulators very greedy in memory.
312 Unfortunately, "auto" is historically the only option available
313 in OpenDaylight CI configuration.
314 To palliate this problem, `tox.ini` current configuration uses the `depends` parameter
315 to artifically chain tests profiles and limit to only 2 the number of controller instances
318 Also, the default display will change from the classical sequence mode.
319 You need to use the option `-o` to get it back (with the notable difference
320 that displayed messages are now mixed between tests profiles).
321 This is the default configuration in the CI::
325 Running different tests in parallel also creates concurrency access problems
326 to others ressources than RAM, mostly the ports to listen to, and the log files.
327 To this sake, lighty.io and karaf build configuration have been customized to change
328 thier listening ports and log files from shell environment variables.
329 This environment variables are also understood by Python tests scripts and tox.
331 You can take a look at the following Gerrit changes for more details
333 https://git.opendaylight.org/gerrit/q/topic:%2522parallel%2522+project:transportpce
337 https://git.opendaylight.org/gerrit/c/transportpce/+/96696
341 https://git.opendaylight.org/gerrit/c/transportpce/+/96662
345 https://git.opendaylight.org/gerrit/c/transportpce/+/96663
347 In a nutshell, TransportPCE CI uses tox and parrallelization to perform functional tests.
348 And if your computer environment has enough RAM and CPU cores,
349 it is perfectly possible to run the same way several functional tests concurrently on it.
350 For example, the following command will test the portmapping behavior
351 for every OpenROADM devices supported versions::
353 $ tox -p 3 -e buildcontroller,sims121,sims221,sims71,tests121,tests221,tests71 portmapping
355 or with karaf alternate builds::
357 $ tox -p 3 -e buildcontroller,build_karaf_tests121,build_karaf_tests221,build_karaf_tests71,sims121,sims221,sims71,tests121,tests221,tests71 portmapping