1 == Backup-Restore test support library ==
3 === 1. Introduction ===
5 The purpose of this library is to allow the generic verification
6 of feature correctness in backup + restore scenarios.
8 A feature is correct from a backup+restore standpoint when,
9 at any provisioning point, a controller backup, followed by a
10 restore, can be performed, and the execution of that procedure
11 will not have any impact on traffic tests or datastore state
12 check in respect to the expected behaviour when no backup + restore
13 procedures are performed.
15 The library can also be used (with minimal modifications) to check
16 whether a feature is safe (continues operating correctly) in the
17 event of a controller reboot (e.g. SFC is known not to, because of
18 keeping certain information (rendered service paths) in the
19 operational DS only and being unable to reconstruct that information
22 === 2. Library usage ===
24 ==== 2.1. Use as Robot Library keywords ====
26 The library is delivered as a readily-available Robot FW library in
27 the ODL integration/test repository. It provides two keywords:
29 - A new keyword ('''BackupRestoreCheck'''), which:
31 # Performs a complete datastore export (using Daexim export rpc)
32 # Does a backup, then a restore of the backup previously
33 created. NOTE: this step is purposefully not implemented in the
34 keyword (a placeholder for concrete backup & restore scripts is
35 provided instead). ODL does not provide a comprehensive B&R implementation:
36 such implementation shall include both the datastore and certain
37 configuration files, but those configuration files depend on the
38 concrete ODL distribution / deployment, so ODL provides only some
39 pieces to implement that backup (i.e. the datastore backup RPCs).
40 This library is contributed with the purpose of easing the testing
41 of any backup&restore implementation; therefore that implementation
42 shall be incorporated to this library (by modifying this step)
43 # Performs a new datastore export
44 # Compares both config & operational datastores for differences (that
45 is, pre-backup and post-restore exports for both datastores), optionally
46 prefiltering those exports using pre-filter files (those prefilter files
47 are passed as parameters to the keyword)
48 # Fails when pre-backup and post-restore exports are different even after
49 removing the specified filtered parts
51 - A new keyword ('''ConditionalBackupRestoreCheck'''), which performs
52 the same steps than BackupRestoreCheck only when a command-line flag ("-v
53 BR_TESTING_ENABLED:true") is present. This allows to easily add
54 backup-restore verification on existing tests, allowing to toggle
55 the execution of that verification
57 ===== 2.1.1. Adding the br verification keyword to an existing robot test =====
59 The library has been designed from the ground up to allow its use in
60 existing testcases, so specific feature provisioning can be tested for
61 correctness in backup-restore scenarios. The design premises for the
63 * To be very easy to incorporate into existing testcases (just one resource
64 import + the verification keyword, that shall be inserted in the existing
65 testcase just after test specific provisioning and before existing test
67 * To be togglable (that is, to allow whether to execute / to bypass the
68 export + backup + restore + export + exports comparison block)
70 ===== 2.1.2. Steps to add backup-restore verification to an existing test suite =====
71 1) Suite setup: Add ClusterManagement Setup (it is needed for Daexim export
72 to work). Hint: if the testsuite already contains an init suite keyword, you
73 can use the Run Keywords construct in order to run both initialization keywords.
75 (Subsequent examples use diff-format):
78 Documentation Test suite for SFC Service Functions, Operates functions from Restconf APIs.
79 -Suite Setup Init Suite
80 +Suite Setup Run Keywords Init Suite ClusterManagement Setup
81 Suite Teardown Delete All Sessions
83 2) Import Backup-Restore support library
85 Resource ../../../libraries/TemplatedRequests.robot
86 +Resource ../../../libraries/BackupRestoreKeywords.robot
88 3) Add the verification keyword (after provisioning, before assertions/traffic verification)
89 Should Contain ${ALLOWED_STATUS_CODES} ${resp.status_code}
90 ${elements}= Create List SFC1-100-Path-1 "parent-service-function-path":"SFC1-100" "hop-number":0 "service-index":255 "ho
91 ... "service-index":254 "hop-number":2 "service-index":253
92 + ConditionalBackupRestoreCheck
93 Check For Elements At URI ${OPERATIONAL_RSPS_URI} ${elements}
95 4) Execute the suite without passing the enablement flag (or pass it disabled:
96 '-v BR_TESTING_ENABLED:false'). Note how the testcase runs as always
97 5) Execute the suite, now passing the enablement flag ('-v BR_TESTING_ENABLED:true'):
98 5.1. If the testcases pass, that means the suite provisioning is safe for B&R
99 (that is, both config and operational DSs are identical before and after the
100 procedure, and any assert / traffic verification the cases perform are also correct.
101 5.2. If a testcase fail:
102 5.2.1. If it is the ConditionalBackupRestoreCheck keyword what fails: it means
103 differences are found in the datastores before the backup / after the restore. Test
104 log should include the list of differences found. Two types of differences:
105 5.2.1.1. If differences are non-issues (e.g. elements whose changes are expected
106 after a backup + restore, as elements containing timestamps that are recalculated
107 after the restore, or elements showing transitory states which are not important
108 regarding B&R correctness), then create as many pre-filter entries as necessary
109 in the corresponding prefilter file (4 prefilter files can be passed to the
110 keyword: prefilter for the config DS before the backup, config DS after restore,
111 operational DS before the backup and operational DS after the restore. Repeat until
112 all unimportant DS entries are filtered
113 5.2.1.2. Differences found on which the former rule is not applicable should be
114 checked carefully, as they are likely to showcase application bugs (regading B&R /
116 5.2.2 Errors in the testcase execution when the BR_TESTING_ENABLED:true flag is
117 passed, in keywords other than the verification keyword, are also candidates to
118 point to application bugs (e.g. because of using runtime-required in-memory
119 state that they fail to reconstruct after the restore), thus requiring careful revision
121 ==== 2.2. Execution as a standalone commandline utility ====
122 In scenarios where Robot FW is not used for testing, the library core (this is, the
123 prefiltered json comparison) can also be used from the command line. The tool is
124 provided as a python commandline utility. Help follows:
126 odluser@odluser-VirtualBox:~/odl/test/csit/libraries/backuprestore\> python JsonDiffTool.py -h
127 usage: JsonDiffTool.py [-h] -i INITIALFILE -f FINALFILE [-ipf INITIALPREFILTER] [-fpf FINALPREFILTER] [-pd] [-v]
128 both initial and final json files are compared for differences. The program
129 returns 0 when the json contents are the same, or the number of differences
130 otherwise. Both json files can be prefiltered for certain patterns before
131 checking the differences
133 -h, --help show this help message and exit
134 -i INITIALFILE, --initialFile INITIALFILE
136 -f FINALFILE, --finalFile FINALFILE
138 -ipf INITIALPREFILTER, --initialPreFilter INITIALPREFILTER
139 File with pre-filtering patterns to apply to the
140 initial json file before comparing
141 -fpf FINALPREFILTER, --finalPreFilter FINALPREFILTER
142 File with pre-filtering patterns to apply to the final
143 json file before comparing
144 -pd, --printDifferences
145 on differences found, prints the list of paths for the
146 found differences before exitting
147 -v, --verbose generate log information
149 ===== 2.2.1. Command-line usage examples =====
150 - Checking for differences between two json files (showing only the number of differences)
151 odluser@odluser-VirtualBox:~/odl/test/csit/libraries/backuprestore\> python JsonDiffTool.py -i ./testinput/arrayTwoNames.json -f ./testinput/arrayThreeNamesSorted.json
154 - Checking for differences and displaying the differences (jsonpatch format)
155 odluser@odluser-VirtualBox:~/odl/test/csit/libraries/backuprestore\> python JsonDiffTool.py -i ./testinput/arrayTwoNames.json -f ./testinput/arrayThreeNamesSorted.json -pd
156 {"path": "/2", "value": {"Name": "Tom"}, "op": "add"}
159 - Checking for differences (and displaying them), using a pre-filter file for the initial json file
160 odluser@odluser-VirtualBox:~/odl/test/csit/libraries/backuprestore\> python JsonDiffTool.py -i ./testinput/mainTestCase/odl_backup_operational_before.json -f testinput/mainTestCase/odl_backup_operational_after.json -ipf testinput/mainTestCase/json_prefilter.conf -pd
161 {"path": "/entity-owners:entity-owners/entity-type/2", "op": "remove"}
162 {"path": "/entity-owners:entity-owners/entity-type/4", "value": {"type": "iface", "entity": [{"owner": "member-1", "id": "/general-entity:entity[general-entity:name='iface']", "candidate": [{"name": "member-1"}]}]}, "op": "add"}
163 {"path": "/network-topology:network-topology/topology/3", "value": {"node": [{"netconf-node-topology:host": "127.0.0.1", "netconf-node-topology:port": 1830, "netconf-node-topology:connection-status": "connecting", "node-id": "CONTROLLER1"}, {"netconf-node-topology:host": "127.0.0.1", "netconf-node-topology:port": 1830, "netconf-node-topology:connection-status": "connecting", "node-id": "CONTROLLER2"}], "topology-id": "topology-netconf"}, "op": "replace"}
164 {"path": "/ietf-yang-library:modules-state/module-set-id", "value": "3", "op": "replace"}
165 {"path": "/ietf-yang-library:modules-state/module/56", "op": "remove"}
166 {"path": "/ietf-yang-library:modules-state/module/116", "op": "remove"}
167 {"path": "/ietf-yang-library:modules-state/module/127", "op": "remove"}
168 {"path": "/ietf-yang-library:modules-state/module/140", "op": "remove"}
169 {"path": "/ietf-yang-library:modules-state/module/139", "op": "remove"}
170 {"path": "/ietf-yang-library:modules-state/module/185", "op": "remove"}
171 {"path": "/ietf-yang-library:modules-state/module/238", "op": "remove"}
172 {"path": "/ietf-yang-library:modules-state/module/267", "op": "remove"}
173 {"path": "/ietf-yang-library:modules-state/module/269", "op": "remove"}
174 {"path": "/ietf-yang-library:modules-state/module/278", "op": "remove"}
175 {"path": "/ietf-yang-library:modules-state/module/277", "op": "remove"}
178 ===== 2.2.2. Unit tests =====
179 A handful of unit tests (testing both jsonpath - jsonpatch expression transformation,
180 difference evaluation, use of filters and error cases) are provided. They can be
181 executed as standard python unittests from the commandline:
183 odluser@odluser-VirtualBox:~/odl/test/csit/libraries/backuprestore\> python backuprestoretest.py
186 .usage: backuprestoretest.py [-h] -i INITIALFILE -f FINALFILE
187 [-ipf INITIALPREFILTER] [-fpf FINALPREFILTER]
189 backuprestoretest.py: error: argument -i/--initialFile is required
200 ----------------------------------------------------------------------
201 Ran 11 tests in 0.881s
204 === 3. Prefilter file format ===
206 * Can contain any number of jsonpath expressions ([http://goessner.net/articles/JsonPath/ jsonpath expressions specification])
207 * Use "#" as line prefix for comments
210 # Pre-filter file example (removes the module from ietf-yang-library:modules-state which name is 'extension-resync-message')
212 $.ietf-yang-library:modules-state.module[?(@.name=='extension-resync-message')]
213 # $.ietf-yang-library:modules-state.module[?(@.name=='extension-switchfeatures-message')]
215 === 4. Dependencies / discarded alternatives ===
217 The library includes in the commit itself the jsonpath library by Phil Budne
218 (https://pypi.python.org/pypi/jsonpath/). This had to be done in order to rename
219 the module (from jsonpath to jsonpathl), because RIDE fails to import a class
220 from a module with the same name, which is the case for this library. The library
221 license (MIT) allows for including / modifying it, so this inclusion is safe license-wise.
223 Other alternatives to this library were explored, but were found unfit for the
224 purpose. Specifically, we tried to use the popular jsonpath-rw library, but it
225 does not support json query filtering by attribute values (only by field names),
226 which is a must. Also objectpath, but the library does not support the return of
227 the path of matching objects (only matched objects themselves). Those paths are
228 required (they are the ones which are transformed into jsonpatch expressions)
230 jsonpatch library (https://pypi.python.org/pypi/jsonpatch) is expected to be
231 installed in order for this library to work. The library is used for removing json