2 Documentation Karaf library. General utility keywords for interacting with the karaf environment, such as the
3 ... karaf console, karaf.log, karaf features, and karaf config files.
5 ... This library is useful to deal with controller Karaf console for ssh sessions in cluster.
6 ... Running Setup_Karaf_Keywords is necessary. If SetupUtils initialization is called, this gets initialized as well.
7 ... If this gets initialized, ClusterManagement gets initialized as well.
9 Library OperatingSystem
10 Library ${CURDIR}/netvirt/excepts.py
11 Resource ${CURDIR}/ClusterManagement.robot
12 Resource ${CURDIR}/SSHKeywords.robot
13 Variables ${CURDIR}/../variables/Variables.py
17 ${connection_index_dict} &{EMPTY}
21 [Arguments] ${http_timeout}=${DEFAULT_TIMEOUT_HTTP}
22 [Documentation] Initialize ClusterManagement. Open ssh karaf connections to each ODL.
23 ClusterManagement.ClusterManagement_Setup http_timeout=${http_timeout}
24 ClusterManagement.Run_Bash_Command_On_List_Or_All iptables -I INPUT -p tcp --dport ${KARAF_SHELL_PORT} -j ACCEPT; iptables-save
25 BuiltIn.Comment First connections to Karaf console may fail, so WUKS is used. TODO: Track as a Bug.
26 FOR ${index} IN @{ClusterManagement__member_index_list}
27 BuiltIn.Run_Keyword_And_Ignore_Error BuiltIn.Wait_Until_Keyword_Succeeds 3s 1s Open_Controller_Karaf_Console_On_Background member_index=${index}
30 Verify_Feature_Is_Installed
31 [Arguments] ${feature_name} ${controller}=${ODL_SYSTEM_IP} ${karaf_port}=${KARAF_SHELL_PORT}
32 [Documentation] Will Succeed if the given ${feature_name} is found in the output of "feature:list -i"
33 ${output} = Issue_Command_On_Karaf_Console feature:list -i | grep ${feature_name} ${controller} ${karaf_port}
34 BuiltIn.Should_Contain ${output} ${feature_name}
37 Issue_Command_On_Karaf_Console
38 [Arguments] ${cmd} ${controller}=${ODL_SYSTEM_IP} ${karaf_port}=${KARAF_SHELL_PORT} ${timeout}=10 ${loglevel}=INFO
39 [Documentation] Will execute the given ${cmd} by ssh'ing to the karaf console running on ${controller}
40 ... Note that this keyword will open&close new SSH connection, without switching back to previously current session.
41 SSHLibrary.Open_Connection ${controller} port=${karaf_port} prompt=${KARAF_PROMPT_LOGIN} timeout=${timeout}
42 SSHLibrary.Login ${KARAF_USER} ${KARAF_PASSWORD} loglevel=${loglevel}
43 SSHLibrary.Write ${cmd}
44 ${output} SSHLibrary.Read_Until_Regexp ${KARAF_PROMPT}
45 SSHLibrary.Write logout
46 SSHLibrary.Close_Connection
50 Safe_Issue_Command_On_Karaf_Console
51 [Arguments] ${cmd} ${controller}=${ODL_SYSTEM_IP} ${karaf_port}=${KARAF_SHELL_PORT} ${timeout}=10 ${loglevel}=INFO
52 [Documentation] Run Issue_Command_On_Karaf_Console but restore previous connection afterwards.
53 BuiltIn.Run_Keyword_And_Return SSHKeywords.Run_Keyword_Preserve_Connection Issue_Command_On_Karaf_Console ${cmd} ${controller} ${karaf_port} ${timeout}
56 Check For Elements On Karaf Command Output Message
57 [Arguments] ${cmd} ${elements} ${controller}=${ODL_SYSTEM_IP} ${karaf_port}=${KARAF_SHELL_PORT} ${timeout}=5
58 [Documentation] Will execute the command using Issue Command On Karaf Console then check for the given elements
59 ... in the command output message
60 ${output} Issue_Command_On_Karaf_Console ${cmd} ${controller} ${karaf_port} ${timeout}
61 FOR ${i} IN @{elements}
62 BuiltIn.Should_Contain ${output} ${i}
65 Verify_Bundle_Is_Installed
66 [Arguments] ${bundle_name} ${controller}=${ODL_SYSTEM_IP} ${karaf_port}=${KARAF_SHELL_PORT}
67 [Documentation] Will succeed if the given ${bundle name} is present in the output of "bundle:list -s "
68 ${output} = Issue_Command_On_Karaf_Console bundle:list -s | grep ${bundle_name} ${controller} ${karaf_port}
69 BuiltIn.Should_Contain ${output} ${bundle_name}
72 Verify_Bundle_Is_Not_Installed
73 [Arguments] ${bundle_name} ${controller}=${ODL_SYSTEM_IP} ${karaf_port}=${KARAF_SHELL_PORT}
74 [Documentation] Will succeed if the given ${bundle_name} is NOT found in the output of "bundle:list -s"
75 ${output} = Issue_Command_On_Karaf_Console bundle:list -i | grep ${bundle_name} ${controller} ${karaf_port}
76 BuiltIn.Should_Not_Contain ${output} ${bundle_name}
79 Check_Karaf_Log_Has_Messages
80 [Arguments] ${filter_string} @{message_list}
81 [Documentation] Will succeed if the @{messages} are found in \ the output of "log:display"
82 ${output} = Issue_Command_On_Karaf_Console log:display | grep ${filter_string}
83 FOR ${message} IN @{message_list}
84 BuiltIn.Should_Contain ${output} ${message}
88 Check_Karaf_Log_Message_Count
89 [Arguments] ${message} ${count} ${use_console}=False
90 [Documentation] Verifies that the ${message} exists specified number of times in
91 ... karaf console log or Karaf Log Folder based on the arg ${use_console}.
92 Run Keyword If ${use_console} == False Check_Karaf_Log_File ${message} ${count}
93 ... ELSE Check_Karaf_Log_From_Console ${message} ${count}
95 Check_Karaf_Log_From_Console
96 [Arguments] ${message} ${count}
97 [Documentation] Verifies that the ${message} exists in the Karaf Console log:display and checks
98 ... that it appears ${count} number of times
99 ${output} = Issue_Command_On_Karaf_Console log:display | grep ${message} | wc -l
100 ${line} = Get Line ${output} 0
101 ${stripped} = Strip String ${line}
102 Should Be Equal As Strings ${stripped} ${count}
105 [Arguments] ${message} ${count}
106 [Documentation] Verifies that the ${message} exists in the Karaf Log Folder and checks
107 ... that it appears ${count} number of times
108 ${output} Run Command On Controller ${ODL_SYSTEM_IP} grep -o ${message} ${WORKSPACE}/${BUNDLEFOLDER}/data/log/* | wc -l
109 Should Be Equal As Strings ${output} ${count}
112 [Arguments] ${feature_name} ${controller}=${ODL_SYSTEM_IP} ${karaf_port}=${KARAF_SHELL_PORT} ${timeout}=180
113 [Documentation] Will Install the given ${feature_name}
114 BuiltIn.Log ${timeout}
115 ${output} = Issue_Command_On_Karaf_Console feature:install ${feature_name} ${controller} ${karaf_port} ${timeout}
116 BuiltIn.Log ${output}
119 Install_A_Feature_Using_Active_Connection
120 [Arguments] ${feature_name}
121 [Documentation] Will Install the given ${feature_name} using active connection
122 ${cmd} = BuiltIn.Set_Variable feature:install ${feature_name}
123 SSHLibrary.Write ${cmd}
124 ${output} SSHLibrary.Read_Until_Regexp ${KARAF_PROMPT}
125 BuiltIn.Log ${output}
129 [Arguments] ${feature_name} ${controller}=${ODL_SYSTEM_IP} ${karaf_port}=${KARAF_SHELL_PORT} ${timeout}=180
130 [Documentation] Will UnInstall the given ${feature_name}
131 ${output} = Issue_Command_On_Karaf_Console feature:uninstall ${feature_name} ${controller} ${karaf_port} ${timeout}
132 BuiltIn.Log ${output}
135 Open_Controller_Karaf_Console_On_Background
136 [Arguments] ${member_index}=${1} ${timeout}=10 ${loglevel}=INFO
137 [Documentation] If there is a stored ssh connection index of connection to the controller's karaf console for ${member_index},
138 ... close the previous connection. In any case create a new connection
139 ... to karaf console for ${member_index}, set correct prompt set and login to karaf console.
140 ... Store connection index for ${member_index} and restore the previous active connection.
141 ${current_ssh_connection_object}= SSHLibrary.Get_Connection
142 BuiltIn.Log ${connection_index_dict}
143 BuiltIn.Log ${member_index}
144 ${status} ${old_connection_index} = BuiltIn.Run_Keyword_And_Ignore_Error Get From Dictionary ${connection_index_dict} ${member_index}
145 BuiltIn.Run_Keyword_If '${status}'=='PASS' BuiltIn.Run_Keywords SSHLibrary.Switch_Connection ${old_connection_index}
146 ... AND BuiltIn.Run_Keyword_And_Ignore_Error SSHLibrary.Write logout
147 ... AND BuiltIn.Run_Keyword_And_Ignore_Error SSHLibrary.Close_Connection
148 ${odl_ip} = ClusterManagement.Resolve_IP_Address_For_Member ${member_index}
149 SSHLibrary.Open_Connection ${odl_ip} port=${KARAF_SHELL_PORT} prompt=${KARAF_PROMPT_LOGIN} timeout=${timeout}
150 ${karaf_connection_object} = SSHLibrary.Get_Connection
151 Collections.Set_To_Dictionary ${connection_index_dict} ${member_index} ${karaf_connection_object.index}
152 SSHLibrary.Login ${KARAF_USER} ${KARAF_PASSWORD} loglevel=${loglevel}
153 [Teardown] SSHKeywords.Restore_Current_Ssh_Connection_From_Index ${current_ssh_connection_object.index}
155 Open_Controller_Karaf_Console_With_Timeout
156 [Arguments] ${member_index}=${1} ${timeout}=3s
157 [Documentation] Open new connection to karaf console for member index with specified timeout.
158 BuiltIn.Log ${member_index}
159 ${odl_ip} = ClusterManagement.Resolve_IP_Address_For_Member ${member_index}
160 SSHLibrary.Open_Connection ${odl_ip} port=${KARAF_SHELL_PORT} prompt=${KARAF_PROMPT_LOGIN} timeout=${timeout}
161 SSHLibrary.Login ${KARAF_USER} ${KARAF_PASSWORD}
163 Configure_Timeout_For_Karaf_Console
164 [Arguments] ${timeout} ${member_index_list}=${EMPTY}
165 [Documentation] Configure a different timeout for each Karaf console.
166 ${index_list} = ClusterManagement.List_Indices_Or_All given_list=${member_index_list}
167 ${current_connection_object} = SSHLibrary.Get_Connection
168 FOR ${member_index} IN @{index_list} # usually: 1, 2, 3
169 ${karaf_connection_index} = Collections.Get_From_Dictionary ${connection_index_dict} ${member_index}
170 SSHLibrary.Switch_Connection ${karaf_connection_index}
171 SSHLibrary.Set_Client_Configuration timeout=${timeout}
173 [Teardown] SSHKeywords.Restore_Current_Ssh_Connection_From_Index ${current_connection_object.index}
175 Execute_Controller_Karaf_Command_On_Background
176 [Arguments] ${command} ${member_index}=${1}
177 [Documentation] Send command to karaf without affecting current SSH connection. Read, log and return response.
178 ${karaf_connection_index} = Collections.Get_From_Dictionary ${connection_index_dict} ${member_index}
179 ${current_connection_index} = SSHLibrary.Switch_Connection ${karaf_connection_index}
180 ${status_write} ${message_write} = BuiltIn.Run_Keyword_And_Ignore_Error SSHLibrary.Write ${command}
181 ${status_wait} ${message_wait} = BuiltIn.Run_Keyword_And_Ignore_Error SSHLibrary.Read_Until_Regexp ${KARAF_PROMPT}
182 BuiltIn.Run Keyword If '${status_write}' != 'PASS' BuiltIn.Fail Failed to send the command: ${command}
183 BuiltIn.Log ${message_wait}
184 BuiltIn.Run_Keyword_If '${status_wait}' != 'PASS' BuiltIn.Fail Failed to see prompt after sending the command: ${command}
185 [Teardown] SSHKeywords.Restore_Current_Ssh_Connection_From_Index ${current_connection_index}
186 [Return] ${message_wait}
188 Execute_Controller_Karaf_Command_With_Retry_On_Background
189 [Arguments] ${command} ${member_index}=${1}
190 [Documentation] Attemp to send command to karaf for ${member_index}, if fail then open connection and try again.
191 ${status} ${message} = BuiltIn.Run_Keyword_And_Ignore_Error Execute_Controller_Karaf_Command_On_Background ${command} ${member_index}
192 BuiltIn.Return_From_Keyword_If '${status}' == 'PASS' ${message}
193 # TODO: Verify this does not leak connections indices.
194 Open_Controller_Karaf_Console_On_Background ${member_index}
195 ${message} = Execute_Controller_Karaf_Command_On_Background ${command} ${member_index}
198 Log_Message_To_Controller_Karaf
199 [Arguments] ${message} ${member_index_list}=${EMPTY} ${tolerate_failure}=True
200 [Documentation] Make sure this resource is initialized. Send a message into the controller's karaf log file on every node listed (or all).
201 ... By default, failure while processing a node is silently ignored, unless ${tolerate_failure} is False.
202 ${index_list} = ClusterManagement.List_Indices_Or_All given_list=${member_index_list}
203 FOR ${index} IN @{index_list} # usually: 1, 2, 3.
204 ${status} ${output} = BuiltIn.Run_Keyword_And_Ignore_Error Execute_Controller_Karaf_Command_With_Retry_On_Background log:log "ROBOT MESSAGE: ${message}" member_index=${index}
205 BuiltIn.Run_Keyword_Unless ${tolerate_failure} or "${status}" == "PASS" BuiltIn.Fail ${output}
208 Log_Test_Suite_Start_To_Controller_Karaf
209 [Arguments] ${member_index_list}=${EMPTY}
210 [Documentation] Log suite name to karaf log, useful in suite setup.
211 Log_Message_To_Controller_Karaf Starting suite ${SUITE_SOURCE} ${member_index_list}
213 Log_Testcase_Start_To_Controller_Karaf
214 [Arguments] ${member_index_list}=${EMPTY}
215 [Documentation] Log test case name to karaf log, useful in test case setup.
216 Log_Message_To_Controller_Karaf Starting test ${SUITE_NAME}.${TEST_NAME} ${member_index_list}
218 Set_Bgpcep_Log_Levels
219 [Arguments] ${bgpcep_level}=${DEFAULT_BGPCEP_LOG_LEVEL} ${protocol_level}=${DEFAULT_PROTOCOL_LOG_LEVEL} ${member_index_list}=${EMPTY}
220 [Documentation] Assuming OCKCOB was used, set logging level on bgpcep and protocol loggers without affecting current SSH session.
221 # FIXME: Move to appropriate Resource
222 ${index_list} = ClusterManagement.List_Indices_Or_All given_list=${member_index_list}
223 FOR ${index} IN @{index_list} # usually: 1, 2, 3.
224 Execute_Controller_Karaf_Command_On_Background log:set ${bgpcep_level} org.opendaylight.bgpcep member_index=${index}
225 Execute_Controller_Karaf_Command_On_Background log:set ${protocol_level} org.opendaylight.protocol member_index=${index}
228 Get Karaf Log Lines From Test Start
229 [Arguments] ${ip} ${test_name} ${cmd} ${user}=${ODL_SYSTEM_USER} ${password}=${ODL_SYSTEM_PASSWORD} ${prompt}=${ODL_SYSTEM_PROMPT}
230 ... ${log_file}=${KARAF_LOG}
231 [Documentation] Scrapes all log messages that match regexp ${type} which fall after a point given by a log message that
232 ... contains ${test_name}. This is useful if your test cases are marking karaf.log with a message indicating when
233 ... that test case has started; such that you can easily pull out any extra log messsages to parse/log/etc in the
234 ... test logic itself. For example, you can grab all ERRORS that occur during your test case.
235 ${output} = Run Command On Controller ${ip} ${cmd} ${user} ${password} ${prompt}
236 @{log_lines} = Split String ${output} ${\n}
237 [Return] ${log_lines}
239 Fail If Exceptions Found During Test
240 [Arguments] ${test_name} ${log_file}=${KARAF_LOG} ${fail}=False
241 [Documentation] Create a failure if an Exception is found in the karaf.log that has not been whitelisted.
242 ... Will work for single controller jobs as well as 3node cluster jobs
243 FOR ${i} IN RANGE 1 ${NUM_ODL_SYSTEM} + 1
244 ${cmd} = Set Variable sed '1,/ROBOT MESSAGE: Starting test ${test_name}/d' ${log_file}
245 ${output} = Get Karaf Log Lines From Test Start ${ODL_SYSTEM_${i}_IP} ${test_name} ${cmd}
246 ${exlist} ${matchlist} = Verify Exceptions ${output}
247 Write Exceptions Map To File ${SUITE_NAME}.${TEST_NAME} /tmp/odl${i}_exceptions.txt
248 ${listlength} = BuiltIn.Get Length ${exlist}
249 BuiltIn.Run Keyword If "${fail}"=="True" and ${listlength} != 0 Log And Fail Exceptions ${exlist} ${listlength}
250 ... ELSE Collections.Log List ${matchlist}
253 Log And Fail Exceptions
254 [Arguments] ${exlist} ${listlength}
255 [Documentation] Print the list of failed exceptions and fail the test
256 Collections.Log List ${exlist}
257 ${exstr} = BuiltIn.Catenate ${exlist}
258 BuiltIn.Fail New exceptions found: ${listlength}\n${exstr}
260 Get Karaf Log Type From Test Start
261 [Arguments] ${ip} ${test_name} ${type} ${user}=${ODL_SYSTEM_USER} ${password}=${ODL_SYSTEM_PASSWORD} ${prompt}=${ODL_SYSTEM_PROMPT}
262 ... ${log_file}=${KARAF_LOG}
263 [Documentation] Scrapes all log messages that match regexp ${type} which fall after a point given by a log message that
264 ... contains ${test_name}. This is useful if your test cases are marking karaf.log with a message indicating when
265 ... that test case has started; such that you can easily pull out any extra log messsages to parse/log/etc in the
266 ... test logic itself. For example, you can grab all ERRORS that occur during your test case.
267 ${cmd} Set Variable sed '1,/ROBOT MESSAGE: Starting test ${test_name}/d' ${log_file} | grep '${type}'
268 ${output} Run Command On Controller ${ip} ${cmd} ${user} ${password} ${prompt}
271 Get Karaf Log Types From Test Start
272 [Arguments] ${ip} ${test_name} ${types} ${user}=${ODL_SYSTEM_USER} ${password}=${ODL_SYSTEM_PASSWORD} ${prompt}=${ODL_SYSTEM_PROMPT}
273 ... ${log_file}=${KARAF_LOG}
274 [Documentation] A wrapper keyword for "Get Karaf Log Type From Test Start" so that we can parse for multiple types
275 ... of log messages. For example, we can grab all messages of type WARN and ERROR
276 FOR ${type} IN @{types}
277 Get Karaf Log Type From Test Start ${ip} ${test_name} ${type} ${user} ${password}
278 ... ${prompt} ${log_file}
281 Get Karaf Log Events From Test Start
282 [Arguments] ${test_name} ${user}=${ODL_SYSTEM_USER} ${password}=${ODL_SYSTEM_PASSWORD} ${prompt}=${ODL_SYSTEM_PROMPT}
283 [Documentation] Wrapper for the wrapper "Get Karaf Log Types From Test Start" so that we can easily loop over
284 ... any number of controllers to analyze karaf.log for ERROR, WARN and Exception log messages
285 ${log_types} = Create List ERROR WARN Exception
286 FOR ${i} IN RANGE 1 ${NUM_ODL_SYSTEM} + 1
287 Get Karaf Log Types From Test Start ${ODL_SYSTEM_${i}_IP} ${test_name} ${log_types}
290 Fail If Exceptions Found During Test Deprecated
291 [Arguments] ${test_name} ${exceptions_white_list}=${EMPTY}
292 [Documentation] Create a failure if an Exception is found in the karaf.log. Will work for single controller jobs
293 ... as well as 3node cluster jobs
294 FOR ${i} IN RANGE 1 ${NUM_ODL_SYSTEM} + 1
295 Verify Exception Logging In Controller ${ODL_SYSTEM_${i}_IP} ${test_name} ${exceptions_white_list}
298 Verify Exception Logging In Controller
299 [Arguments] ${controller_ip} ${test_name} ${exceptions_white_list}
300 [Documentation] Local keyword to make it easier to loop through N controllers to pull Exceptions from the
301 ... karaf.log file and validate with "Check Against White List"
302 ${exceptions}= Get Karaf Log Type From Test Start ${controller_ip} ${test_name} Exception
303 @{log_lines}= Split String ${exceptions} ${\n}
304 ${num_log_entries} Get Length ${log_lines}
305 Return From Keyword If ${num_log_entries} == ${0} No Exceptions found.
306 FOR ${log_message} IN @{log_lines}
307 Check Against White List ${log_message} ${exceptions_white_list}
310 Check Against White List
311 [Arguments] ${exception_line} ${exceptions_white_list}
312 [Documentation] As soon as the ${exceptions_line} is found in one of the elements of ${exceptions_white_list}
313 ... this keyword will exit and give a Pass to the caller. If there is no match, this keyword will end up
314 ... marking a failure. In the case that no exceptions are found, the caller could end up passing a single
315 ... empty line as that is what is returned when a grep on karaf.log has no match, so we can safely return
316 ... in that case as well.
317 Return From Keyword If "${exception_line}" == ""
318 FOR ${exception} IN @{exceptions_white_list}
319 Return From Keyword If "${exception}" in "${exception_line}" Exceptions found, but whitelisted: ${\n}${exception_line}${\n}
321 Fail Exceptions Found: ${\n}${exception_line}${\n}
324 [Arguments] ${message} ${timeout}=60 ${member_index}=${1}
325 [Documentation] Read karaf logs until message appear
326 # TODO: refactor this keyword to use the new workflow to account for multiple controllers. Initial work was done
327 # in this patch https://git.opendaylight.org/gerrit/#/c/45596/
328 # however, the consumers of this keyword were breaking after that change. Initial theory is that a previous
329 # keyword used before this "Wait For Karaf Log" keyword was closing the karaf console connection, so the
330 # "Flexible SSH Login" keyword from the patch above (45596) was failing.
331 BuiltIn.Log Waiting for '${message}' in karaf log
332 SSHLibrary.Open_Connection ${ODL_SYSTEM_IP} port=${KARAF_SHELL_PORT} prompt=${KARAF_PROMPT_LOGIN} timeout=${timeout}
333 SSHLibrary.Login ${KARAF_USER} ${KARAF_PASSWORD} loglevel=${loglevel}
334 SSHLibrary.Write log:tail
335 SSHLibrary.Read_Until ${message}
336 SSHLibrary.Write logout
337 SSHLibrary.Close_Connection
340 [Arguments] ${bundle_id}
341 [Documentation] Restarts bundle passed as argument. Note this operation is only for testing and not production environments
342 # TODO: prepare this for cluster environment and multiple controllers
343 Execute_Controller_Karaf_Command_With_Retry_On_Background bundle:restart $(bundle:id '${bundle_id}')
346 [Documentation] Restarts Karaf and polls log to detect when Karaf is up and running again
347 # TODO: prepare this for cluster environment and multiple controllers
348 Execute_Controller_Karaf_Command_With_Retry_On_Background log:clear
349 Execute_Controller_Karaf_Command_With_Retry_On_Background shutdown -r -f
350 BuiltIn.Run_Keyword_And_Return_Status BuiltIn.Wait_Until_Keyword_Succeeds 240s 60s Wait_For_Karaf_Log Shiro environment initialized in
353 [Documentation] Restarts jetty bundle (to reload certificates or key/truststore information)
354 Execute_Controller_Karaf_Command_With_Retry_On_Background log:clear
355 Restart_Bundle OPS4J Pax Web - Jetty
356 Wait_For_Karaf_Log Started jetty-default