+++ /dev/null
-== Backup-Restore test support library ==
-
-=== 1. Introduction ===
-
-The purpose of this library is to allow the generic verification
-of feature correctness in backup + restore scenarios.
-
-A feature is correct from a backup+restore standpoint when,
-at any provisioning point, a controller backup, followed by a
-restore, can be performed, and the execution of that procedure
-will not have any impact on traffic tests or datastore state
-check in respect to the expected behaviour when no backup + restore
-procedures are performed.
-
-The library can also be used (with minimal modifications) to check
-whether a feature is safe (continues operating correctly) in the
-event of a controller reboot (e.g. SFC is known not to, because of
-keeping certain information (rendered service paths) in the
-operational DS only and being unable to reconstruct that information
-after a reboot).
-
-=== 2. Library usage ===
-
-==== 2.1. Use as Robot Library keywords ====
-
-The library is delivered as a readily-available Robot FW library in
-the ODL integration/test repository. It provides two keywords:
-
-- A new keyword ('''BackupRestoreCheck'''), which:
-
-# Performs a complete datastore export (using Daexim export rpc)
-# Does a backup, then a restore of the backup previously
- created. NOTE: this step is purposefully not implemented in the
- keyword (a placeholder for concrete backup & restore scripts is
- provided instead). ODL does not provide a comprehensive B&R implementation:
- such implementation shall include both the datastore and certain
- configuration files, but those configuration files depend on the
- concrete ODL distribution / deployment, so ODL provides only some
- pieces to implement that backup (i.e. the datastore backup RPCs).
- This library is contributed with the purpose of easing the testing
- of any backup&restore implementation; therefore that implementation
- shall be incorporated to this library (by modifying this step)
-# Performs a new datastore export
-# Compares both config & operational datastores for differences (that
- is, pre-backup and post-restore exports for both datastores), optionally
- prefiltering those exports using pre-filter files (those prefilter files
- are passed as parameters to the keyword)
-# Fails when pre-backup and post-restore exports are different even after
- removing the specified filtered parts
-
-- A new keyword ('''ConditionalBackupRestoreCheck'''), which performs
- the same steps than BackupRestoreCheck only when a command-line flag ("-v
- BR_TESTING_ENABLED:true") is present. This allows to easily add
-backup-restore verification on existing tests, allowing to toggle
-the execution of that verification
-
-===== 2.1.1. Adding the br verification keyword to an existing robot test =====
-
-The library has been designed from the ground up to allow its use in
-existing testcases, so specific feature provisioning can be tested for
-correctness in backup-restore scenarios. The design premises for the
-library were:
-* To be very easy to incorporate into existing testcases (just one resource
- import + the verification keyword, that shall be inserted in the existing
- testcase just after test specific provisioning and before existing test
-verification steps)
-* To be togglable (that is, to allow whether to execute / to bypass the
-export + backup + restore + export + exports comparison block)
-
-===== 2.1.2. Steps to add backup-restore verification to an existing test suite =====
-1) Suite setup: Add ClusterManagement Setup (it is needed for Daexim export
- to work). Hint: if the testsuite already contains an init suite keyword, you
- can use the Run Keywords construct in order to run both initialization keywords.
-
-(Subsequent examples use diff-format):
-
-*** Settings ***
-Documentation Test suite for SFC Service Functions, Operates functions from Restconf APIs.
--Suite Setup Init Suite
-+Suite Setup Run Keywords Init Suite ClusterManagement Setup
-Suite Teardown Delete All Sessions
-
-2) Import Backup-Restore support library
-
-Resource ../../../libraries/TemplatedRequests.robot
-+Resource ../../../libraries/BackupRestoreKeywords.robot
-
-3) Add the verification keyword (after provisioning, before assertions/traffic verification)
-Should Contain ${ALLOWED_STATUS_CODES} ${resp.status_code}
- ${elements}= Create List SFC1-100-Path-1 "parent-service-function-path":"SFC1-100" "hop-number":0 "service-index":255 "ho
-... "service-index":254 "hop-number":2 "service-index":253
-+ ConditionalBackupRestoreCheck
-Check For Elements At URI ${OPERATIONAL_RSPS_URI} ${elements}
-
-4) Execute the suite without passing the enablement flag (or pass it disabled:
- '-v BR_TESTING_ENABLED:false'). Note how the testcase runs as always
-5) Execute the suite, now passing the enablement flag ('-v BR_TESTING_ENABLED:true'):
-5.1. If the testcases pass, that means the suite provisioning is safe for B&R
- (that is, both config and operational DSs are identical before and after the
- procedure, and any assert / traffic verification the cases perform are also correct.
-5.2. If a testcase fail:
-5.2.1. If it is the ConditionalBackupRestoreCheck keyword what fails: it means
- differences are found in the datastores before the backup / after the restore. Test
-log should include the list of differences found. Two types of differences:
-5.2.1.1. If differences are non-issues (e.g. elements whose changes are expected
- after a backup + restore, as elements containing timestamps that are recalculated
- after the restore, or elements showing transitory states which are not important
- regarding B&R correctness), then create as many pre-filter entries as necessary
- in the corresponding prefilter file (4 prefilter files can be passed to the
- keyword: prefilter for the config DS before the backup, config DS after restore,
- operational DS before the backup and operational DS after the restore. Repeat until
- all unimportant DS entries are filtered
-5.2.1.2. Differences found on which the former rule is not applicable should be
- checked carefully, as they are likely to showcase application bugs (regading B&R /
- controller reboots)
-5.2.2 Errors in the testcase execution when the BR_TESTING_ENABLED:true flag is
- passed, in keywords other than the verification keyword, are also candidates to
- point to application bugs (e.g. because of using runtime-required in-memory
- state that they fail to reconstruct after the restore), thus requiring careful revision
-
-==== 2.2. Execution as a standalone commandline utility ====
-In scenarios where Robot FW is not used for testing, the library core (this is, the
-prefiltered json comparison) can also be used from the command line. The tool is
-provided as a python commandline utility. Help follows:
-
- odluser@odluser-VirtualBox:~/odl/test/csit/libraries/backuprestore\> python JsonDiffTool.py -h
- usage: JsonDiffTool.py [-h] -i INITIALFILE -f FINALFILE [-ipf INITIALPREFILTER] [-fpf FINALPREFILTER] [-pd] [-v]
- both initial and final json files are compared for differences. The program
- returns 0 when the json contents are the same, or the number of differences
- otherwise. Both json files can be prefiltered for certain patterns before
- checking the differences
- optional arguments:
- -h, --help show this help message and exit
- -i INITIALFILE, --initialFile INITIALFILE
- initial json file
- -f FINALFILE, --finalFile FINALFILE
- final json file
- -ipf INITIALPREFILTER, --initialPreFilter INITIALPREFILTER
- File with pre-filtering patterns to apply to the
- initial json file before comparing
- -fpf FINALPREFILTER, --finalPreFilter FINALPREFILTER
- File with pre-filtering patterns to apply to the final
- json file before comparing
- -pd, --printDifferences
- on differences found, prints the list of paths for the
- found differences before exitting
- -v, --verbose generate log information
-
-===== 2.2.1. Command-line usage examples =====
-- Checking for differences between two json files (showing only the number of differences)
- odluser@odluser-VirtualBox:~/odl/test/csit/libraries/backuprestore\> python JsonDiffTool.py -i ./testinput/arrayTwoNames.json -f ./testinput/arrayThreeNamesSorted.json
- 1
-
-- Checking for differences and displaying the differences (jsonpatch format)
- odluser@odluser-VirtualBox:~/odl/test/csit/libraries/backuprestore\> python JsonDiffTool.py -i ./testinput/arrayTwoNames.json -f ./testinput/arrayThreeNamesSorted.json -pd
- {"path": "/2", "value": {"Name": "Tom"}, "op": "add"}
- 1
-
-- Checking for differences (and displaying them), using a pre-filter file for the initial json file
- odluser@odluser-VirtualBox:~/odl/test/csit/libraries/backuprestore\> python JsonDiffTool.py -i ./testinput/mainTestCase/odl_backup_operational_before.json -f testinput/mainTestCase/odl_backup_operational_after.json -ipf testinput/mainTestCase/json_prefilter.conf -pd
- {"path": "/entity-owners:entity-owners/entity-type/2", "op": "remove"}
- {"path": "/entity-owners:entity-owners/entity-type/4", "value": {"type": "iface", "entity": [{"owner": "member-1", "id": "/general-entity:entity[general-entity:name='iface']", "candidate": [{"name": "member-1"}]}]}, "op": "add"}
- {"path": "/network-topology:network-topology/topology/3", "value": {"node": [{"netconf-node-topology:host": "127.0.0.1", "netconf-node-topology:port": 1830, "netconf-node-topology:connection-status": "connecting", "node-id": "CONTROLLER1"}, {"netconf-node-topology:host": "127.0.0.1", "netconf-node-topology:port": 1830, "netconf-node-topology:connection-status": "connecting", "node-id": "CONTROLLER2"}], "topology-id": "topology-netconf"}, "op": "replace"}
- {"path": "/ietf-yang-library:modules-state/module-set-id", "value": "3", "op": "replace"}
- {"path": "/ietf-yang-library:modules-state/module/56", "op": "remove"}
- {"path": "/ietf-yang-library:modules-state/module/116", "op": "remove"}
- {"path": "/ietf-yang-library:modules-state/module/127", "op": "remove"}
- {"path": "/ietf-yang-library:modules-state/module/140", "op": "remove"}
- {"path": "/ietf-yang-library:modules-state/module/139", "op": "remove"}
- {"path": "/ietf-yang-library:modules-state/module/185", "op": "remove"}
- {"path": "/ietf-yang-library:modules-state/module/238", "op": "remove"}
- {"path": "/ietf-yang-library:modules-state/module/267", "op": "remove"}
- {"path": "/ietf-yang-library:modules-state/module/269", "op": "remove"}
- {"path": "/ietf-yang-library:modules-state/module/278", "op": "remove"}
- {"path": "/ietf-yang-library:modules-state/module/277", "op": "remove"}
- 15
-
-===== 2.2.2. Unit tests =====
-A handful of unit tests (testing both jsonpath - jsonpatch expression transformation,
-difference evaluation, use of filters and error cases) are provided. They can be
-executed as standard python unittests from the commandline:
-
- odluser@odluser-VirtualBox:~/odl/test/csit/libraries/backuprestore\> python backuprestoretest.py
- 0
- .2
- .usage: backuprestoretest.py [-h] -i INITIALFILE -f FINALFILE
- [-ipf INITIALPREFILTER] [-fpf FINALPREFILTER]
- [-pd] [-v]
- backuprestoretest.py: error: argument -i/--initialFile is required
- .14
- 14
- .16
- 16
- .16
- 16
- .1
- ..16
- 16
- ...
- ----------------------------------------------------------------------
- Ran 11 tests in 0.881s
- OK
-
-=== 3. Prefilter file format ===
-Prefilter files:
-* Can contain any number of jsonpath expressions ([http://goessner.net/articles/JsonPath/ jsonpath expressions specification])
-* Use "#" as line prefix for comments
-Example:
- #
- # Pre-filter file example (removes the module from ietf-yang-library:modules-state which name is 'extension-resync-message')
- #
- $.ietf-yang-library:modules-state.module[?(@.name=='extension-resync-message')]
- # $.ietf-yang-library:modules-state.module[?(@.name=='extension-switchfeatures-message')]
-
-=== 4. Dependencies / discarded alternatives ===
-
-The library includes in the commit itself the jsonpath library by Phil Budne
-(https://pypi.python.org/pypi/jsonpath/). This had to be done in order to rename
-the module (from jsonpath to jsonpathl), because RIDE fails to import a class
-from a module with the same name, which is the case for this library. The library
-license (MIT) allows for including / modifying it, so this inclusion is safe license-wise.
-
-Other alternatives to this library were explored, but were found unfit for the
-purpose. Specifically, we tried to use the popular jsonpath-rw library, but it
-does not support json query filtering by attribute values (only by field names),
-which is a must. Also objectpath, but the library does not support the return of
-the path of matching objects (only matched objects themselves). Those paths are
-required (they are the ones which are transformed into jsonpatch expressions)
-
-jsonpatch library (https://pypi.python.org/pypi/jsonpatch) is expected to be
-installed in order for this library to work. The library is used for removing json
-elements via patches
Code Review / integration / test.git / blobdiff