2 Documentation Suite for testing upgrading persisted data from earlier release.
4 ... Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.
6 ... This program and the accompanying materials are made available under the
7 ... terms of the Eclipse Public License v1.0 which accompanies this distribution,
8 ... and is available at http://www.eclipse.org/legal/epl-v10.html
11 ... This suite kills the running (newer) ODL at its default location.
12 ... It then installs (configurable) older ODL to an alternative location,
13 ... pushes large amount of car data, verifies and kills the older ODL.
14 ... The journal and snapshot files are transferred to the default location
15 ... and the newer ODL is started.
16 ... Then it verifies the config data is still present and matches what was seen before.
18 ... In principle, the suite should also work if "newer" ODL is in fact older.
19 ... The limiting factor is featuresBoot, the value should be applicable to both ODL versions.
21 ... Note that in order to create traffic large enough for snapshots to be created,
22 ... this suite also actis as a stress test for Restconf.
23 ... But as that is not a primary focus of this suite,
24 ... data seen on newer ODL is only compared to what was seen on the older ODL
25 ... (stored in ${data_before} suite variable).
27 ... As using Robotframework would be both too slow and too memory consuming,
28 ... this suite uses a specialized Python utility for pushing the data locally on ODL_SYSTEM.
29 ... The utility filename is configurable, as there may be changes in PATCH behavior in future.
31 ... This suite uses relatively new support for PATCH http method.
32 ... It repetitively replaces a segment of cars with moving IDs,
33 ... so that there is a lot of data in journal (both write and delete),
34 ... but the overall size of data stored remains limited.
36 ... This is 1-node suite, but it still uses ClusterManagement.Check_Cluster_Is_In_Sync
37 ... in order to detect the same sync condition as 3-node suite would do.
38 ... Jolokia feature is required for that.
40 ... Minimal set of features to be installed: odl-restconf, odl-jolokia, odl-clustering-test-app.
41 Suite Setup Setup_Suite
42 Test Setup SetupUtils.Setup_Test_With_Logging_And_Without_Fast_Failing
43 Test Teardown SetupUtils.Teardown_Test_Show_Bugs_If_Test_Failed
44 Default Tags 1node carpeople critical
47 Resource ${CURDIR}/../../../libraries/CompareStream.robot
48 Resource ${CURDIR}/../../../libraries/ClusterManagement.robot
49 Resource ${CURDIR}/../../../libraries/SetupUtils.robot
50 Resource ${CURDIR}/../../../libraries/SSHKeywords.robot
51 Resource ${CURDIR}/../../../libraries/TemplatedRequests.robot
52 Resource ${CURDIR}/../../../libraries/NexusKeywords.robot
55 ${ALTERNATIVE_BUNDLEFOLDER_PARENT} /tmp/older
56 ${CAR_VAR_DIR} ${CURDIR}/../../../variables/carpeople/libtest/cars
57 ${CLUSTER_BOOTUP_SYNC_TIMEOUT} 1200s # Rebooting after kill may take longer time, especially for -all- install.
60 ${PREVIOUS_ODL_RELEASE_ZIP_URL} ${EMPTY}
61 ${PYTHON_UTILITY_FILENAME} replace_cars.py
65 Select_Latest_Previous_Release_If_Not_Specified
66 [Documentation] If previous ODL release is not specified, then latest release for previous to current stream is used.
67 ... Note: If current stream is not found on nexus, then it is taken as new one (not released yet).
68 ... So in this case, latest release version is return.
69 BuiltIn.Set_Suite_Variable ${odl_zip_url} ${PREVIOUS_ODL_RELEASE_ZIP_URL}
70 BuiltIn.Pass_Execution_If '${odl_zip_url}'!='' Bundle url to previous release is provided: ${PREVIOUS_ODL_RELEASE_ZIP_URL}, no need to search in nexus.
71 ${previous_release} = NexusKeywords.Get_Latest_ODL_Previous_Stream_Release_URL ${ODL_STREAM}
72 BuiltIn.Set_Suite_Variable ${odl_zip_url} ${previous_release}
75 [Documentation] The ODL prepared by releng/builder is the newer one, kill it.
76 ... Also, remove journal and snapshots.
77 ClusterManagement.Kill_Members_From_List_Or_All
78 # This is deliberately analoguous to killing the whole cluster.
79 # (As opposed to killing just one member, but for 1 node it is the same.)
80 ClusterManagement.Clean_Journals_Data_And_Snapshots_On_List_Or_All
83 [Documentation] Download .zip of older ODL, unpack, delete .zip, copy featuresBoot line.
85 SSHKeywords.Execute_Command_Should_Pass mkdir -p "${ALTERNATIVE_BUNDLEFOLDER_PARENT}" && cd "${ALTERNATIVE_BUNDLEFOLDER_PARENT}" && rm -rf * && wget -N "${odl_zip_url}"
86 # Unzip and detect bundle folder name.
87 ${bundle_dir} = SSHKeywords.Execute_Command_Should_Pass cd "${ALTERNATIVE_BUNDLEFOLDER_PARENT}" && unzip -q *.zip && rm *.zip && ls -1
88 BuiltIn.Set_Suite_Variable \${alternative_bundlefolder} ${ALTERNATIVE_BUNDLEFOLDER_PARENT}/${bundle_dir}
89 # TODO: Add more strict checks. Folder should have single line, without .zip extension.
90 # Extract featuresBoot lines.
91 ${cfg_filename} = BuiltIn.Set_Variable org.apache.karaf.features.cfg
92 ${cfg_older} = BuiltIn.Set_Variable ${alternative_bundlefolder}/etc/${cfg_filename}
93 ${cfg_newer} = BuiltIn.Set_Variable ${WORKSPACE}/${BUNDLEFOLDER}/etc/${cfg_filename}
94 ${vanilla_line} = SSHKeywords.Execute_Command_Should_Pass grep 'featuresBoot' "${cfg_older}" | grep -v 'featuresBootAsynchronous'
95 ${csit_line} = SSHKeywords.Execute_Command_Should_Pass grep 'featuresBoot' "${cfg_newer}" | grep -v 'featuresBootAsynchronous'
96 ${karaf4_features} = Extract_Karaf4_Boot_Features ${csit_line}
97 # TODO: this works only if old odl is karaf3. Once old and new odl both are e.g. karaf4 this will create invalid line.
98 ${new_csit_line} = CompareStream.Set_Variable_If_At_Least_Nitrogen ${vanilla_line},${karaf4_features} ${csit_line}
99 # Replace the vanilla line.
100 SSHKeywords.Execute_Command_Should_Pass sed -i 's/${vanilla_line}/${new_csit_line}/g' "${cfg_older}"
101 # Verify the replaced line.
102 ${updated_line} = SSHKeywords.Execute_Command_Should_Pass grep 'featuresBoot' "${cfg_older}" | grep -v 'featuresBootAsynchronous'
103 BuiltIn.Should_Not_Be_Equal ${vanilla_line} ${updated_line}
104 BuiltIn.Should_Be_Equal ${new_csit_line} ${updated_line}
107 [Documentation] Start older ODL on background.
108 [Tags] 1node carpeople # Not critical, to save space in default log.html presentation
109 ClusterManagement.Start_Members_From_List_Or_All wait_for_sync=True timeout=${CLUSTER_BOOTUP_SYNC_TIMEOUT} karaf_home=${alternative_bundlefolder}
112 [Documentation] Put car data to config datastore of older ODL.
113 ${command} = BuiltIn.Set_Variable python ${PYTHON_UTILITY_FILENAME} --segment-size=${SEGMENT_SIZE} --iterations=${ITERATIONS} --move-per-iter=${MOVE_PER_ITER}
114 SSHKeywords.Execute_Command_Should_Pass ${command} stderr_must_be_empty=True
115 # TODO: I have seen 401 here once. Implement workaround or report a Bug.
118 [Documentation] Get and save the stored data for later comparison.
119 ${data} = TemplatedRequests.Get_As_Json_Templated folder=${CAR_VAR_DIR} verify=False
120 BuiltIn.Set_Suite_Variable \${data_before} ${data}
123 [Documentation] Compare the saved data against what the data should look like.
124 ${first_id} = BuiltIn.Evaluate (${ITERATIONS} - 1) * ${MOVE_PER_ITER} + 1
125 SetupUtils.Set_Known_Bug_Id 5909
126 # The following line is the second part of TemplatedRequests.Get_As_Json_Templated for verify=True.
127 TemplatedRequests.Verify_Response_As_Json_Templated response=${data_before} folder=${CAR_VAR_DIR} base_name=data iterations=${SEGMENT_SIZE} iter_start=${first_id}
130 [Documentation] Kill the older ODL immediatelly.
131 ClusterManagement.Kill_Members_From_List_Or_All
133 Transfer_Persisted_Data
134 [Documentation] Move snapshots and (segmented-)journal into the original ODL installation.
135 # SSHLibrary.Switch_Connection ${odl_system_ssh_index}
136 ${stdout} = SSHKeywords.Execute_Command_Should_Pass cp -rv "${alternative_bundlefolder}/snapshots" "${WORKSPACE}/${BUNDLEFOLDER}/" && cp -rv "${alternative_bundlefolder}/*journal" "${WORKSPACE}/${BUNDLEFOLDER}/"
137 Set_Suite_Variable ${stdout}
139 Check_Snapshot_With_Transferred_Persisted_Data
140 [Documentation] Fail if no snapshot file was created.
141 BuiltIn.Should_Contain ${stdout} /snapshots/ msg='Snapshot file was not created.'
144 [Documentation] Start the newer ODL on background.
145 [Tags] 1node carpeople # Not critical, to save space in default log.html presentation
146 ClusterManagement.Start_Members_From_List_Or_All wait_for_sync=True timeout=${CLUSTER_BOOTUP_SYNC_TIMEOUT}
148 Verify_Data_Is_Restored
149 [Documentation] Wait until data check succeeds (there may be temporary 503).
150 BuiltIn.Wait_Until_Keyword_Succeeds 180s 5s Check_Restored_Data
152 Archive_Older_Karaf_Log
153 [Documentation] Only original location benefits from automatic karaf.log archivation.
154 SSHKeywords.Execute_Command_Should_Pass xz -9evv ${alternative_bundlefolder}/data/log/karaf.log
155 SSHLibrary.Get_File ${alternative_bundlefolder}/data/log/karaf.log.xz older.karaf.log.xz
156 # TODO: Uncompress first (or last) megabyte for better readability?
160 [Documentation] Activate dependency Resources, create SSH connection, copy Python utility.
161 SetupUtils.Setup_Utils_For_Setup_And_Teardown
162 TemplatedRequests.Create_Default_Session
163 ${connection} = SSHKeywords.Open_Connection_To_ODL_System
164 SSHLibrary.Put_File ${CURDIR}/../../../../tools/odl-mdsal-clustering-tests/${PYTHON_UTILITY_FILENAME}
167 [Documentation] Get car data from config datastore and check it is equal to the stored data.
168 ... This has to be a separate keyword, as it is run under WUKS.
169 ${data_after} = TemplatedRequests.Get_As_Json_Templated folder=${CAR_VAR_DIR} verify=False
170 BuiltIn.Should_Be_Equal ${data_before} ${data_after}
172 Extract_Karaf4_Boot_Features
173 [Arguments] ${csit_line}
174 [Documentation] Extract boot features. It is used for the 1st line of karaf4 featuresBoot parameter from org.apache.karaf.features.cfg.
175 ${bfeatures} String.Replace_String ${csit_line} ${Space} ${Empty}
176 ${bfeatures} String.Replace_String ${bfeatures} featuresBoot= ${Empty}
177 ${bfeatures} String.Replace_String ${bfeatures} ,\\ ${Empty}
178 BuiltIn.Return_From_Keyword ${bfeatures}