2 Documentation Suite for testing upgrading persisted data from earlier release.
4 ... Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.
6 ... This program and the accompanying materials are made available under the
7 ... terms of the Eclipse Public License v1.0 which accompanies this distribution,
8 ... and is available at http://www.eclipse.org/legal/epl-v10.html
11 ... This suite kills the running (newer) ODL at its default location.
12 ... It then installs (configurable) older ODL to an alternative location,
13 ... pushes large amount of car data, verifies and kills the older ODL.
14 ... The journal and snapshot files are transferred to the default location
15 ... and the newer ODL is started.
16 ... Then it verifies the config data is still present and matches what was seen before.
18 ... In principle, the suite should also work if "newer" ODL is in fact older.
19 ... The limiting factor is featuresBoot, the value should be applicable to both ODL versions.
21 ... Note that in order to create traffic large enough for snapshots to be created,
22 ... this suite also actis as a stress test for Restconf.
23 ... But as that is not a primary focus of this suite,
24 ... data seen on newer ODL is only compared to what was seen on the older ODL
25 ... (stored in ${data_before} suite variable).
27 ... As using Robotframework would be both too slow and too memory consuming,
28 ... this suite uses a specialized Python utility for pushing the data locally on ODL_SYSTEM.
29 ... The utility filename is configurable, as there may be changes in PATCH behavior in future.
31 ... This suite uses relatively new support for PATCH http method.
32 ... It repetitively replaces a segment of cars with moving IDs,
33 ... so that there is a lot of data in journal (both write and delete),
34 ... but the overall size of data stored remains limited.
36 ... This is 1-node suite, but it still uses ClusterManagement.Check_Cluster_Is_In_Sync
37 ... in order to detect the same sync condition as 3-node suite would do.
38 ... Jolokia feature is required for that.
40 ... Minimal set of features to be installed: odl-restconf, odl-jolokia, odl-clustering-test-app.
44 Resource ${CURDIR}/../../../libraries/ClusterManagement.robot
45 Resource ${CURDIR}/../../../libraries/SetupUtils.robot
46 Resource ${CURDIR}/../../../libraries/SSHKeywords.robot
47 Resource ${CURDIR}/../../../libraries/TemplatedRequests.robot
48 Resource ${CURDIR}/../../../libraries/NexusKeywords.robot
50 Suite Setup Setup_Suite
51 Test Setup SetupUtils.Setup_Test_With_Logging_And_Without_Fast_Failing
52 Test Teardown SetupUtils.Teardown_Test_Show_Bugs_If_Test_Failed
54 Default Tags 1node carpeople critical
58 ${ALTERNATIVE_BUNDLEFOLDER_PARENT} /tmp/older
59 ${CAR_VAR_DIR} ${CURDIR}/../../../variables/carpeople/libtest/cars
60 # Rebooting after kill may take longer time, especially for -all- install.
61 ${CLUSTER_BOOTUP_SYNC_TIMEOUT}
65 ${PREVIOUS_ODL_RELEASE_ZIP_URL} ${EMPTY}
66 ${PYTHON_UTILITY_FILENAME} replace_cars.py
71 Select_Latest_Previous_Release_If_Not_Specified
72 [Documentation] If previous ODL release is not specified, then latest release for previous to current stream is used.
73 ... Note: If current stream is not found on nexus, then it is taken as new one (not released yet).
74 ... So in this case, latest release version is return.
75 BuiltIn.Set_Suite_Variable ${odl_zip_url} ${PREVIOUS_ODL_RELEASE_ZIP_URL}
76 BuiltIn.Pass_Execution_If
77 ... '${odl_zip_url}'!=''
78 ... Bundle url to previous release is provided: ${PREVIOUS_ODL_RELEASE_ZIP_URL}, no need to search in nexus.
79 ${previous_release} = NexusKeywords.Get_Latest_ODL_Previous_Stream_Release_URL ${ODL_STREAM}
80 BuiltIn.Set_Suite_Variable ${odl_zip_url} ${previous_release}
83 [Documentation] The ODL prepared by releng/builder is the newer one, kill it.
84 ... Also, remove journal and snapshots.
85 ClusterManagement.Kill_Members_From_List_Or_All
86 # This is deliberately analoguous to killing the whole cluster.
87 # (As opposed to killing just one member, but for 1 node it is the same.)
88 ClusterManagement.Clean_Journals_Data_And_Snapshots_On_List_Or_All
91 [Documentation] Download .zip of older ODL, unpack, delete .zip, copy featuresBoot line.
93 SSHKeywords.Execute_Command_Should_Pass
94 ... mkdir -p "${ALTERNATIVE_BUNDLEFOLDER_PARENT}" && cd "${ALTERNATIVE_BUNDLEFOLDER_PARENT}" && rm -rf * && wget -N "${odl_zip_url}"
95 # Unzip and detect bundle folder name.
96 ${bundle_dir} = SSHKeywords.Execute_Command_Should_Pass
97 ... cd "${ALTERNATIVE_BUNDLEFOLDER_PARENT}" && unzip -q *.zip && rm *.zip && ls -1
98 BuiltIn.Set_Suite_Variable \${alternative_bundlefolder} ${ALTERNATIVE_BUNDLEFOLDER_PARENT}/${bundle_dir}
99 # TODO: Add more strict checks. Folder should have single line, without .zip extension.
100 # Extract featuresBoot lines.
101 ${cfg_filename} = BuiltIn.Set_Variable org.apache.karaf.features.cfg
102 ${cfg_older} = BuiltIn.Set_Variable ${alternative_bundlefolder}/etc/${cfg_filename}
103 ${cfg_newer} = BuiltIn.Set_Variable ${WORKSPACE}/${BUNDLEFOLDER}/etc/${cfg_filename}
104 ${vanilla_line} = SSHKeywords.Execute_Command_Should_Pass
105 ... grep 'featuresBoot' "${cfg_older}" | grep -v 'featuresBootAsynchronous'
106 ${csit_line} = SSHKeywords.Execute_Command_Should_Pass
107 ... grep 'featuresBoot' "${cfg_newer}" | grep -v 'featuresBootAsynchronous'
108 ${karaf4_features} = Extract_Karaf4_Boot_Features ${csit_line}
109 # TODO: this works only if old odl is karaf3. Once old and new odl both are e.g. karaf4 this will create invalid line.
110 ${new_csit_line} = ${vanilla_line},${karaf4_features}
111 # Replace the vanilla line.
112 SSHKeywords.Execute_Command_Should_Pass sed -i 's/${vanilla_line}/${new_csit_line}/g' "${cfg_older}"
113 # Verify the replaced line.
114 ${updated_line} = SSHKeywords.Execute_Command_Should_Pass
115 ... grep 'featuresBoot' "${cfg_older}" | grep -v 'featuresBootAsynchronous'
116 BuiltIn.Should_Not_Be_Equal ${vanilla_line} ${updated_line}
117 BuiltIn.Should_Be_Equal ${new_csit_line} ${updated_line}
120 [Documentation] Start older ODL on background.
121 [Tags] 1node carpeople # Not critical, to save space in default log.html presentation
122 ClusterManagement.Start_Members_From_List_Or_All
123 ... wait_for_sync=True
124 ... timeout=${CLUSTER_BOOTUP_SYNC_TIMEOUT}
125 ... karaf_home=${alternative_bundlefolder}
128 [Documentation] Put car data to config datastore of older ODL.
129 ${command} = BuiltIn.Set_Variable
130 ... python ${PYTHON_UTILITY_FILENAME} --segment-size=${SEGMENT_SIZE} --iterations=${ITERATIONS} --move-per-iter=${MOVE_PER_ITER}
131 SSHKeywords.Execute_Command_Should_Pass ${command} stderr_must_be_empty=True
132 # TODO: I have seen 401 here once. Implement workaround or report a Bug.
135 [Documentation] Get and save the stored data for later comparison.
136 ${data} = TemplatedRequests.Get_As_Json_Templated folder=${CAR_VAR_DIR} verify=False
137 BuiltIn.Set_Suite_Variable \${data_before} ${data}
140 [Documentation] Compare the saved data against what the data should look like.
141 ${first_id} = BuiltIn.Evaluate (${ITERATIONS} - 1) * ${MOVE_PER_ITER} + 1
142 SetupUtils.Set_Known_Bug_Id 5909
143 # The following line is the second part of TemplatedRequests.Get_As_Json_Templated for verify=True.
144 TemplatedRequests.Verify_Response_As_Json_Templated
145 ... response=${data_before}
146 ... folder=${CAR_VAR_DIR}
148 ... iterations=${SEGMENT_SIZE}
149 ... iter_start=${first_id}
152 [Documentation] Kill the older ODL immediatelly.
153 ClusterManagement.Kill_Members_From_List_Or_All
155 Transfer_Persisted_Data
156 [Documentation] Move snapshots and (segmented-)journal into the original ODL installation.
157 # SSHLibrary.Switch_Connection ${odl_system_ssh_index}
158 ${stdout} = SSHKeywords.Execute_Command_Should_Pass
159 ... cp -rv "${alternative_bundlefolder}/snapshots" "${WORKSPACE}/${BUNDLEFOLDER}/" && cp -rv "${alternative_bundlefolder}/"*journal "${WORKSPACE}/${BUNDLEFOLDER}/"
160 Set_Suite_Variable ${stdout}
162 Check_Snapshot_With_Transferred_Persisted_Data
163 [Documentation] Fail if no snapshot file was created.
164 BuiltIn.Should_Contain ${stdout} /snapshots/ msg='Snapshot file was not created.'
167 [Documentation] Start the newer ODL on background.
168 [Tags] 1node carpeople # Not critical, to save space in default log.html presentation
169 ClusterManagement.Start_Members_From_List_Or_All wait_for_sync=True timeout=${CLUSTER_BOOTUP_SYNC_TIMEOUT}
171 Verify_Data_Is_Restored
172 [Documentation] Wait until data check succeeds (there may be temporary 503).
173 BuiltIn.Wait_Until_Keyword_Succeeds 180s 5s Check_Restored_Data
175 Archive_Older_Karaf_Log
176 [Documentation] Only original location benefits from automatic karaf.log archivation.
177 SSHKeywords.Execute_Command_Should_Pass xz -9evv ${alternative_bundlefolder}/data/log/karaf.log
178 SSHLibrary.Get_File ${alternative_bundlefolder}/data/log/karaf.log.xz older.karaf.log.xz
179 # TODO: Uncompress first (or last) megabyte for better readability?
184 [Documentation] Activate dependency Resources, create SSH connection, copy Python utility.
185 SetupUtils.Setup_Utils_For_Setup_And_Teardown
186 TemplatedRequests.Create_Default_Session
187 ${connection} = SSHKeywords.Open_Connection_To_ODL_System
188 SSHLibrary.Put_File ${CURDIR}/../../../../tools/odl-mdsal-clustering-tests/${PYTHON_UTILITY_FILENAME}
191 [Documentation] Get car data from config datastore and check it is equal to the stored data.
192 ... This has to be a separate keyword, as it is run under WUKS.
193 ${data_after} = TemplatedRequests.Get_As_Json_Templated folder=${CAR_VAR_DIR} verify=False
194 BuiltIn.Should_Be_Equal ${data_before} ${data_after}
196 Extract_Karaf4_Boot_Features
197 [Documentation] Extract boot features. It is used for the 1st line of karaf4 featuresBoot parameter from org.apache.karaf.features.cfg.
198 [Arguments] ${csit_line}
199 ${bfeatures} = String.Replace_String ${csit_line} ${Space} ${Empty}
200 ${bfeatures} = String.Replace_String ${bfeatures} featuresBoot= ${Empty}
201 ${bfeatures} = String.Replace_String ${bfeatures} ,\\ ${Empty}