2 Documentation Nexus repository access keywords, and supporting Java and Maven handling.
4 ... Copyright (c) 2015,2016 Cisco Systems, Inc. and others. All rights reserved.
6 ... This program and the accompanying materials are made available under the
7 ... terms of the Eclipse Public License v1.0 which accompanies this distribution,
8 ... and is available at http://www.eclipse.org/legal/epl-v10.html
11 ... This library encapsulates a bunch of somewhat complex and commonly used
12 ... Nexus operations into reusable keywords to make writing test suites easier.
14 ... Currently, Java version detection is incorporated so that Java tools can be run reliably.
15 ... Also, suport for installing and running Maven is added, as that needs the Java detection.
16 ... TODO: Move Java detection and Maven to a separate Resource, or rename this Resource.
18 Library OperatingSystem
23 Library RequestsLibrary
24 Resource ${CURDIR}/CompareStream.robot
25 Resource ${CURDIR}/SSHKeywords.robot
26 Resource ${CURDIR}/Utils.robot
29 &{COMPONENT_MAPPING} netconf=netconf-impl bgpcep=pcep-impl carpeople=clustering-it-model yangtools=yang-data-impl bindingv1=mdsal-binding-generator-impl
30 @{RELEASE_INTEGRATED_COMPONENTS} mdsal odlparent yangtools
32 ${JAVA_8_HOME_CENTOS} /usr/lib/jvm/java-1.8.0
33 ${JAVA_8_HOME_UBUNTU} /usr/lib/jvm/java-8-openjdk-amd64
34 ${JAVA_11_HOME_CENTOS} /usr/lib/jvm/java-11-openjdk
35 ${JAVA_11_HOME_UBUNTU} /usr/lib/jvm/java-11-openjdk-amd64
36 ${JAVA_OPTIONS} -Xmx2560m # Note that '-Xmx=3g' is wrong syntax. Also 3GB heap may not fit in 4GB RAM.
37 ${MAVEN_DEFAULT_OUTPUT_FILENAME} default_maven.log
38 ${MAVEN_OPTIONS} -Pq -Djenkins
39 ${MAVEN_REPOSITORY_PATH} /tmp/r
40 ${MAVEN_SETTINGS_URL} https://raw.githubusercontent.com/opendaylight/odlparent/master/settings.xml
41 ${MAVEN_VERSION} 3.3.9
42 ${NEXUS_FALLBACK_URL} ${NEXUSURL_PREFIX}/content/repositories/opendaylight.snapshot
43 ${NEXUS_RELEASE_BASE_URL} https://nexus.opendaylight.org/content/repositories/opendaylight.release
44 ${NEXUS_RELEASES_URL} ${NEXUS_RELEASE_BASE_URL}/org/opendaylight/integration/opendaylight
47 Initialize_Artifact_Deployment_And_Usage
48 [Arguments] ${tools_system_connect}=True
49 [Documentation] Places search utility to ODL system, which will be needed for version detection.
50 ... By default also initialize a SSH connection to Tools system,
51 ... as following Keywords assume a working connection towards target system.
52 # Connect to the ODL machine.
53 ${odl} = SSHKeywords.Open_Connection_To_ODL_System
54 # Deploy the search tool.
55 SSHLibrary.Put_File ${CURDIR}/../../tools/deployment/search.sh
56 SSHLibrary.Close_Connection
57 # Optionally connect to the Tools System machine.
58 BuiltIn.Return_From_Keyword_If not (${tools_system_connect}) # the argument may be a convoluted Python expression
59 SSHKeywords.Open_Connection_To_Tools_System
61 NexusKeywords__Get_Items_To_Look_At
62 [Arguments] ${component}
63 [Documentation] Get a list of items that might contain the version number that we are looking for.
65 ... &{COMPONENT_MAPPING} is the centralized place to maintain the mapping
66 ... from a stream independent component nickname to the list of artifact names to search for.
67 Collections.Dictionary_Should_Contain_Key ${COMPONENT_MAPPING} ${component} Component not supported by NexusKeywords version detection: ${component}
68 BuiltIn.Run_Keyword_And_Return Collections.Get_From_Dictionary ${COMPONENT_MAPPING} ${component}
70 NexusKeywords__Detect_Version_To_Pull
71 [Arguments] ${component}
72 [Documentation] Determine the exact Nexus directory to be used as a source for a particular test tool
73 ... Figure out what version of the tool needs to be pulled out of the
74 ... Nexus by looking at the version directory of the subsystem from
75 ... which the tool is being pulled. This code is REALLY UGLY but there
76 ... is no way around it until the bug
77 ... https://bugs.opendaylight.org/show_bug.cgi?id=5206 gets fixed.
78 ... I also don't want to depend on maven-metadata-local.xml and other
79 ... bits and pieces of ODL distribution which are not required for ODL
80 ... to function properly.
81 ${itemlist} = NexusKeywords__Get_Items_To_Look_At ${component}
82 ${current_ssh_connection} = SSHLibrary.Get Connection
83 SSHKeywords.Open_Connection_To_ODL_System
84 ${version} ${result} = SSHLibrary.Execute_Command sh search.sh ${WORKSPACE}/${BUNDLEFOLDER}/system ${itemlist} return_rc=True
85 SSHLibrary.Close_Connection
86 SSHKeywords.Restore Current SSH Connection From Index ${current_ssh_connection.index}
87 BuiltIn.Log ${version}
88 BuiltIn.Run_Keyword_If ${result}!=0 BuiltIn.Fail Component "${component}": searching for "${itemlist}" found no version, cannot locate test tool.
89 ${version} ${location} = String.Split_String ${version} max_split=1
90 [Return] ${version} ${location}
94 [Documentation] On active SSH conenction execute download ${url} command, log output, check RC and return file name.
95 ${filename} = String.Fetch_From_Right ${url} /
96 ${response} ${result} = SSHLibrary.Execute_Command wget -q -N '${url}' 2>&1 return_rc=True
97 BuiltIn.Log ${response}
98 BuiltIn.Run_Keyword_If ${result} != 0 BuiltIn.Fail File ${filename} could not be downloaded from ${url}
102 [Arguments] ${component} ${artifact} ${name_prefix}=${artifact}- ${name_suffix}=-executable.jar ${fallback_url}=${NEXUS_FALLBACK_URL} ${explicit_url}=${EMPTY}
103 [Documentation] Deploy the specified artifact from Nexus to the cwd of the machine to which the active SSHLibrary connection points.
104 ... ${component} is a name part of an artifact present in system/ of ODl installation with the same version as ${artifact} should have.
105 ... Must have ${BUNDLE_URL} variable set to the URL from which the
106 ... tested ODL distribution was downloaded and this place must be
107 ... inside a repository created by a standard distribution
108 ... construction job. If this is detected to ne be the case, fallback URL is used.
109 ... If ${explicit_url} is non-empty, Deploy_From_Utrl is called instead.
110 ... TODO: Allow deploying to a specific directory, we have SSHKeywords.Execute_Command_At_Cwd_Should_Pass now.
111 BuiltIn.Run_Keyword_And_Return_If """${explicit_url}""" != "" Deploy_From_Url ${explicit_url}
112 ${urlbase} = String.Fetch_From_Left ${BUNDLE_URL} /org/opendaylight
113 # If the BUNDLE_URL points somewhere else (perhaps *patch-test* job in Jenkins),
114 # ${urlbase} is the whole ${BUNDLE_URL}, in which case we use the ${fallback_url}
115 # If we are working with a "release integrated" project, we always will want to look for
116 # a released version, not in the snapshots
117 ${urlbase} = BuiltIn.Set_Variable_If '${urlbase}' != '${BUNDLE_URL}' ${urlbase} ${fallback_url}
118 ${urlbase} = BuiltIn.Set_Variable_If '${component}' in @{RELEASE_INTEGRATED_COMPONENTS} ${NEXUS_RELEASE_BASE_URL} ${urlbase}
119 ${version} ${location} = NexusKeywords__Detect_Version_To_Pull ${component}
120 # TODO: Use RequestsLibrary and String instead of curl and bash utilities?
121 ${url} = BuiltIn.Set_Variable ${urlbase}/${location}/${artifact}/${version}
122 # TODO: Review SSHKeywords for current best keywords to call.
123 ${metadata} = SSHKeywords.Execute_Command_Should_Pass curl -L ${url}/maven-metadata.xml
124 ${status} ${namepart} = BuiltIn.Run_Keyword_And_Ignore_Error SSHKeywords.Execute_Command_Should_Pass echo "${metadata}" | grep value | head -n 1 | cut -d '>' -f 2 | cut -d '<' -f 1 stderr_must_be_empty=${True}
125 ${length} = BuiltIn.Get_Length ${namepart}
126 ${namepart} = BuiltIn.Set_Variable_If "${status}" != "PASS" or ${length} == 0 ${version} ${namepart}
127 ${filename} = BuiltIn.Set_Variable ${name_prefix}${namepart}${name_suffix}
128 BuiltIn.Log ${filename}
129 ${url} = BuiltIn.Set_Variable ${url}/${filename}
130 ${response} ${result} = SSHLibrary.Execute_Command wget -q -N '${url}' 2>&1 return_rc=True
131 BuiltIn.Log ${response}
132 BuiltIn.Run_Keyword_If ${result} != 0 BuiltIn.Fail Artifact "${artifact}" in component "${component}" could not be downloaded from ${url}
136 [Arguments] ${component} ${artifact} ${suffix}=executable ${fallback_url}=${NEXUS_FALLBACK_URL} ${explicit_url}=${EMPTY}
137 [Documentation] Deploy a test tool.
138 ... The test tools have naming convention of the form
139 ... "<repository_url>/some/dir/somewhere/<tool-name>/<tool-name>-<version-tag>-${suffix}.jar"
140 ... where "<tool-name>" is the name of the tool and "<version-tag>" is
141 ... the version tag that is digged out of the maven metadata. This
142 ... keyword calculates ${name_prefix} and ${name_suffix} for
143 ... "Deploy_Artifact" and then calls "Deploy_Artifact" to do the real
144 ... work of deploying the artifact.
145 ${name_prefix} = BuiltIn.Set_Variable ${artifact}-
146 ${name_suffix} = BuiltIn.Set_Variable_If "${suffix}" != "" -${suffix}.jar .jar
147 ${filename} = Deploy_Artifact ${component} ${artifact} ${name_prefix} ${name_suffix} ${fallback_url}
151 Install_And_Start_Java_Artifact
152 [Arguments] ${component} ${artifact} ${suffix}=executable ${tool_options}=${EMPTY} ${java_options}=${EMPTY} ${openjdk}=${JDKVERSION}
153 ... ${fallback_url}=${NEXUS_FALLBACK_URL} ${explicit_url}=${EMPTY}
154 [Documentation] Deploy the artifact, assign name for log file, figure out java command, write the command to active SSH connection and return the log name.
155 ... This keyword does not examine whether the artifact was started successfully or whether is still running upon return.
156 # TODO: Unify this keyword with what NexusKeywords.Install_And_Start_Testtool does.
157 ${actual_java_options} = BuiltIn.Set_Variable_If """${java_options}""" != "" ${java_options} ${JAVA_OPTIONS}
158 ${filename} = Deploy_Test_Tool ${component} ${artifact} ${suffix} ${fallback_url} ${explicit_url}
159 ${command} = Compose_Full_Java_Command ${actual_java_options} -jar ${filename} ${tool_options}
160 ${logfile} = Utils.Get_Log_File_Name ${artifact}
161 SSHLibrary.Write ${command} >${logfile} 2>&1
164 Compose_Dilemma_Filepath
165 [Arguments] ${default_path} ${specific_path}
166 [Documentation] Query active SSH connection, return specific path if it exists else default path.
167 ${out} ${rc} = SSHLibrary.Execute_Command ls -lA ${specific_path} 2>&1 return_rc=True
168 BuiltIn.Return_From_Keyword_If ${rc} == 0 ${specific_path}
169 BuiltIn.Return_From_Keyword ${default_path}
171 Compose_Base_Java_Command
172 [Arguments] ${openjdk}=${JDKVERSION}
173 [Documentation] Return string suitable for launching Java programs over SSHLibrary, depending on JRE version needed.
174 ... This requires that the SSH connection on which the command is going to be used is active as it is needed for querying files.
175 ... Commands composed for one SSH connection shall not be reused on other SSH connections as the two connections may have different Java setups.
176 ... Not directly related to Nexus, but versioned Java tools may need this.
177 # Check whether the user set the override and return it if yes.
178 BuiltIn.Run_Keyword_And_Return_If """${openjdk}""" == "openjdk8" Compose_Dilemma_Filepath ${JAVA_8_HOME_CENTOS}/bin/java ${JAVA_8_HOME_UBUNTU}/bin/java
179 BuiltIn.Run_Keyword_And_Return_If """${openjdk}""" == "openjdk11" Compose_Dilemma_Filepath ${JAVA_11_HOME_CENTOS}/bin/java ${JAVA_11_HOME_UBUNTU}/bin/java
180 # Attempt to call plain "java" command directly. If it works, return it.
181 ${out} ${rc} = SSHLibrary.Execute_Command java -version 2>&1 return_rc=True
182 BuiltIn.Return_From_Keyword_If ${rc} == 0 java
183 # Query the virtual machine for the JAVA_HOME environment variable and
184 # use it to assemble a (hopefully) working command. If that worked out,
186 ${java} = SSHLibrary.Execute_Command echo $JAVA_HOME/bin/java 2>&1
187 ${out} ${rc} = SSHLibrary.Execute_Command ${java} -version 2>&1 return_rc=True
188 BuiltIn.Return_From_Keyword_If ${rc} == 0 ${java}
189 # There are bizzare test environment setups where the (correct) JAVA_HOME
190 # is set in the VM where Robot is running but not in the VM where the
191 # tools are supposed to run (usually because these two are really one
192 # and the same system and idiosyncracies of BASH prevent easy injection
193 # of the JAVA_HOME environment variable into a place where connections
194 # made by SSHLibrary would pick it up). So try to use that value to
195 # create a java command and check that it works.
196 ${JAVA_HOME} = OperatingSystem.Get_Environment_Variable JAVA_HOME ${EMPTY}
197 ${java} = BuiltIn.Set_Variable_If """${JAVA_HOME}"""!="" ${JAVA_HOME}/bin/java false
198 ${out} ${rc} = SSHLibrary.Execute_Command ${java} -version 2>&1 return_rc=True
199 BuiltIn.Return_From_Keyword_If ${rc} == 0 ${java}
200 # Nothing works, most likely java is not installed at all on the target
201 # machine or it is hopelesly lost. Bail out with a helpful message
202 # telling the user how to make it accessible for the script.
203 BuiltIn.Fail Unable to find Java; specify \${JDKVERSION}, put it to your PATH or set JAVA_HOME environment variable.
205 Compose_Full_Java_Command
206 [Arguments] ${options} ${openjdk}=${JDKVERSION}
207 [Documentation] Return full Bash command to run Java with given options.
208 ... This requires that the SSH connection on which the command is going to be used is active as it is needed for querying files.
209 ... The options may include JVM options, application command line arguments, Bash redirects and other constructs.
210 ${base_command} = Compose_Base_Java_Command openjdk=${openjdk}
211 ${full_command} = BuiltIn.Set_Variable ${base_command} ${options}
212 BuiltIn.Log ${full_command}
213 [Return] ${full_command}
216 [Arguments] ${openjdk}=${JDKVERSION}
217 [Documentation] Compose base java command and strip trailing "/bin/java".
218 ${java_command} = Compose_Base_Java_Command
219 ${java_home} ${bin} ${java} = String.Split_String_From_Right ${java_command} separator=/ max_split=2
220 [Return] ${java_home}
223 [Arguments] ${maven_version}=3.3.9 ${openjdk}=${JDKVERSION}
224 [Documentation] Download and unpack Maven, prepare launch command with proper Java version and download settings file.
225 ... This Keyword requires an active SSH connection to target machine.
226 ... This Keyword sets global variables, so that suites can reuse existing installation.
227 ... This Keyword can only place Maven (and settings) to remote current working directory.
228 ... This Keyword does not perform any initial or final cleanup.
229 # Avoid multiple initialization by several downstream libraries.
230 ${installed_version} = BuiltIn.Get_Variable_Value \${Maven__installed_version} None
231 BuiltIn.Return_From_Keyword_If """${installed_version}""" == """${maven_version}"""
232 BuiltIn.Set_Global_Variable \${Maven__installed_version} ${maven_version}
233 BuiltIn.Set_Global_Variable \${maven_directory} apache-maven-${maven_version}
234 SSHKeywords.Execute_Command_At_Cwd_Should_Pass rm -rf '${maven_directory}'
235 ${maven_archive_filename} = BuiltIn.Set_Variable ${maven_directory}-bin.tar.gz
236 ${maven_download_url} = BuiltIn.Set_Variable http://www-us.apache.org/dist/maven/maven-3/${maven_version}/binaries/${maven_archive_filename}
237 SSHKeywords.Execute_Command_At_Cwd_Should_Pass wget -N '${maven_download_url}' stderr_must_be_empty=False
238 SSHKeywords.Execute_Command_At_Cwd_Should_Pass tar xvf '${maven_archive_filename}'
239 ${java_home} = NexusKeywords.Compose_Java_Home openjdk=${openjdk}
240 BuiltIn.Set_Global_Variable \${maven_bash_command} export JAVA_HOME='${java_home}' && export MAVEN_OPTS='${JAVA_OPTIONS}' && ./${maven_directory}/bin/mvn
241 # TODO: Get settings files from Jenkins settings provider, somehow.
242 SSHKeywords.Execute_Command_At_Cwd_Should_Pass wget '${MAVEN_SETTINGS_URL}' -O settings.xml stderr_must_be_empty=False
245 [Arguments] ${maven_version}=3.3.9 ${openjdk}=${JDKVERSION} ${branch}=${EMPTY} ${patches}=${EMPTY}
246 [Documentation] Install Maven.
247 ... Depending on arguments, perform a multipatch build to populate local Maven repository with patched artifacts.
248 Install_Maven_Bare maven_version=${maven_version} openjdk=${openjdk}
249 BuiltIn.Return_From_Keyword_If """${patches}""" == "" No post-install build requested.
250 BuiltIn.Run_Keyword_If """${branch}""" == "" BuiltIn.Fail BRANCH needs to be specified for multipatch builds.
251 ${script_name} = BuiltIn.Set_Variable include-raw-integration-multipatch-distribution-test.sh
252 ${script_url} = BuiltIn.Set_Variable https://raw.githubusercontent.com/opendaylight/releng-builder/master/jjb/integration/${script_name}
253 SSHKeywords.Execute_Command_At_Cwd_Should_Pass wget -N '${script_url}' stderr_must_be_empty=False
254 SSHKeywords.Execute_Command_At_Cwd_Should_Pass export WORKSPACE='${WORKSPACE}' && export BRANCH='${branch}' && export PATCHES_TO_BUILD='${patches}' && bash '${script_name}' stderr_must_be_empty=False
255 Run_Maven pom_file=${WORKSPACE}/patch_tester/pom.xml
258 [Arguments] ${pom_file}=pom.xml ${log_file}=${MAVEN_DEFAULT_OUTPUT_FILENAME}
259 [Documentation] Determine arguments to use and call mvn command against given pom file.
260 SSHKeywords.Execute_Command_At_Cwd_Should_Pass mkdir -p '${MAVEN_REPOSITORY_PATH}'
261 ${maven_repository_options} = BuiltIn.Set_Variable -Dmaven.repo.local=${MAVEN_REPOSITORY_PATH} -Dorg.ops4j.pax.url.mvn.localRepository=${MAVEN_REPOSITORY_PATH}
262 SSHKeywords.Execute_Command_At_Cwd_Should_Pass ${maven_bash_command} clean install dependency:tree -V -B -DoutputFile=dependency_tree.log -s './settings.xml' -f '${pom_file}' ${MAVEN_OPTIONS} ${maven_repository_options} > '${log_file}'
264 Get_ODL_Versions_From_Nexus
265 [Documentation] Returns name of last release found on nexus and list of all versions.
266 RequestsLibrary.Create_Session nexus ${NEXUS_RELEASES_URL} verify=${TRUE}
267 ${uri}= BuiltIn.Set_Variable maven-metadata.xml
268 ${response}= RequestsLibrary.Get_Request nexus ${uri}
269 BuiltIn.Log ${response.text}
270 ${root}= XML.Parse_XML ${response.text}
271 ${element}= XML.Get_Element ${root} versioning/latest
272 ${latest}= BuiltIn.Set_Variable ${element.text}
273 BuiltIn.Log ${latest}
274 @{elements}= XML.Get_Elements ${root} .//version
275 ${versions}= BuiltIn.Create_List
276 FOR ${element} IN @{elements}
277 Collections.Append_To_List ${versions} ${element.text}
279 Collections.Sort_List ${versions}
280 BuiltIn.Log_Many @{versions}
281 [Return] ${latest} @{versions}
283 Get_Latest_ODL_Release_From_Nexus
284 [Documentation] Returns name of last release found on nexus
285 ${latest} @{versions}= Get_ODL_Versions_From_Nexus
288 Get_Latest_ODL_Stream_Release
289 [Arguments] ${stream}=latest
290 [Documentation] Returns name for last release for specified stream.
291 ${latest} @{versions}= Get_ODL_Versions_From_Nexus
292 BuiltIn.Return_From_Keyword_If '${stream}'=='latest' ${latest}
293 ${latest_version}= BuiltIn.Set_Variable xxx
294 FOR ${version} IN @{versions}
295 ${latest_version}= BuiltIn.Set_Variable_If '${stream}'.title() in '${version}' ${version} ${latest_version}
297 BuiltIn.Run_Keyword_If '${latest_version}'=='xxx' BuiltIn.Fail Could not find latest release for stream ${stream}
298 BuiltIn.Log ${latest_version}
299 [Return] ${latest_version}
301 Get_Latest_ODL_Stream_Release_URL
302 [Arguments] ${stream}=latest ${format}=.zip
303 [Documentation] Returns URL for last release for specified stream. Default format is .zip.
304 ${latest_version}= Get_Latest_ODL_Stream_Release ${stream}
305 ${url}= BuiltIn.Set_Variable ${NEXUS_RELEASES_URL}/${latest_version}/opendaylight-${latest_version}${format}
309 Get_Latest_ODL_Previous_Stream_Release
310 [Arguments] ${stream}=${ODL_STREAM}
311 [Documentation] Returns name for last release for previous stream of specified stream.
312 ... Note: If specified stream is not found on nexus, then it is taken as new one (not released yet).
313 ... So in this case, latest release version is return.
315 ... NOTE: the below logic is stripping the initial 0. values from the 0.x.x version string that is
316 ... the current (and future) version numbering scheme. There is always a leading 0. to the version
317 ... strings and stripping it makes is easier to do int comparison to find the largest version in the
318 ... list. Comparing as strings does not work. There are some python libs like distutils.version
319 ... or packaging that can do a better job comparing versions, but since ODL version numbering is simple
320 ... at this point, this convention will suffice. The leading 0. will be added back after the the latest
321 ... version is found from the list. The CompareStream.robot library keeps a mapping of major version
322 ... numbers to the global variable ${ODL_STREAM} so that is used to ensure we get a major version that is
323 ... older than the current running major version.
324 ${latest} @{versions}= Get_ODL_Versions_From_Nexus
325 ${current_version}= BuiltIn.Set_Variable &{Stream_dict}[${ODL_STREAM}].0
326 ${latest_version}= BuiltIn.Set_Variable 0.0
327 FOR ${version} IN @{versions}
328 ${version} = String.Replace String Using Regexp ${version} ^0\. ${EMPTY}
329 ${latest_version}= Set Variable If ${version} > ${latest_version} and ${version} < ${current_version} ${version} ${latest_version}
331 ${latest_version}= Set Variable 0.${latest_version}
332 BuiltIn.Run_Keyword_If '${latest_version}'=='0.0.0' BuiltIn.Fail Could not find latest previous release for stream ${stream}
333 BuiltIn.Log ${latest_version}
334 [Return] ${latest_version}
336 Get_Latest_ODL_Previous_Stream_Release_URL
337 [Arguments] ${stream}=${ODL_STREAM} ${format}=.zip
338 [Documentation] Returns URL for last release for previous stream of specified stream. Default format is .zip.
339 ${latest_version}= Get_Latest_ODL_Previous_Stream_Release ${stream}
340 ${url}= BuiltIn.Set_Variable ${NEXUS_RELEASES_URL}/${latest_version}/opendaylight-${latest_version}${format}