2 Documentation Karaf library. General utility keywords for interacting with the karaf environment, such as the
3 ... karaf console, karaf.log, karaf features, and karaf config files.
5 ... This library is useful to deal with controller Karaf console for ssh sessions in cluster.
6 ... Running Setup_Karaf_Keywords is necessary. If SetupUtils initialization is called, this gets initialized as well.
7 ... If this gets initialized, ClusterManagement gets initialized as well.
9 Library OperatingSystem
10 Resource ${CURDIR}/ClusterManagement.robot
11 Resource ${CURDIR}/SSHKeywords.robot
12 Variables ${CURDIR}/../variables/Variables.py
16 ${connection_index_dict} &{EMPTY}
20 [Arguments] ${http_timeout}=${DEFAULT_TIMEOUT_HTTP}
21 [Documentation] Initialize ClusterManagement. Open ssh karaf connections to each ODL.
22 ClusterManagement.ClusterManagement_Setup http_timeout=${http_timeout}
23 BuiltIn.Comment First connections to Karaf console may fail, so WUKS is used. TODO: Track as a Bug.
24 : FOR ${index} IN @{ClusterManagement__member_index_list}
25 \ BuiltIn.Run_Keyword_And_Ignore_Error BuiltIn.Wait_Until_Keyword_Succeeds 3s 1s Open_Controller_Karaf_Console_On_Background member_index=${index}
27 Verify_Feature_Is_Installed
28 [Arguments] ${feature_name} ${controller}=${ODL_SYSTEM_IP} ${karaf_port}=${KARAF_SHELL_PORT}
29 [Documentation] Will Succeed if the given ${feature_name} is found in the output of "feature:list -i"
30 ${output} = Issue_Command_On_Karaf_Console feature:list -i | grep ${feature_name} ${controller} ${karaf_port}
31 BuiltIn.Should_Contain ${output} ${feature_name}
34 Issue_Command_On_Karaf_Console
35 [Arguments] ${cmd} ${controller}=${ODL_SYSTEM_IP} ${karaf_port}=${KARAF_SHELL_PORT} ${timeout}=10 ${loglevel}=INFO
36 [Documentation] Will execute the given ${cmd} by ssh'ing to the karaf console running on ${controller}
37 ... Note that this keyword will open&close new SSH connection, without switching back to previously current session.
38 SSHLibrary.Open_Connection ${controller} port=${karaf_port} prompt=${KARAF_PROMPT_LOGIN} timeout=${timeout}
39 SSHLibrary.Login ${KARAF_USER} ${KARAF_PASSWORD} loglevel=${loglevel}
40 SSHLibrary.Write ${cmd}
41 ${output} SSHLibrary.Read_Until_Regexp ${KARAF_PROMPT}
42 SSHLibrary.Close_Connection
46 Safe_Issue_Command_On_Karaf_Console
47 [Arguments] ${cmd} ${controller}=${ODL_SYSTEM_IP} ${karaf_port}=${KARAF_SHELL_PORT} ${timeout}=10 ${loglevel}=INFO
48 [Documentation] Run Issue_Command_On_Karaf_Console but restore previous connection afterwards.
49 BuiltIn.Run_Keyword_And_Return SSHKeywords.Run_Keyword_Preserve_Connection Issue_Command_On_Karaf_Console ${cmd} ${controller} ${karaf_port} ${timeout}
52 Check For Elements On Karaf Command Output Message
53 [Arguments] ${cmd} ${elements} ${controller}=${ODL_SYSTEM_IP} ${karaf_port}=${KARAF_SHELL_PORT} ${timeout}=5
54 [Documentation] Will execute the command using Issue Command On Karaf Console then check for the given elements
55 ... in the command output message
56 ${output} Issue_Command_On_Karaf_Console ${cmd} ${controller} ${karaf_port} ${timeout}
57 : FOR ${i} IN @{elements}
58 \ BuiltIn.Should_Contain ${output} ${i}
60 Verify_Bundle_Is_Installed
61 [Arguments] ${bundle_name} ${controller}=${ODL_SYSTEM_IP} ${karaf_port}=${KARAF_SHELL_PORT}
62 [Documentation] Will succeed if the given ${bundle name} is present in the output of "bundle:list -s "
63 ${output} = Issue_Command_On_Karaf_Console bundle:list -s | grep ${bundle_name} ${controller} ${karaf_port}
64 BuiltIn.Should_Contain ${output} ${bundle_name}
67 Verify_Bundle_Is_Not_Installed
68 [Arguments] ${bundle_name} ${controller}=${ODL_SYSTEM_IP} ${karaf_port}=${KARAF_SHELL_PORT}
69 [Documentation] Will succeed if the given ${bundle_name} is NOT found in the output of "bundle:list -s"
70 ${output} = Issue_Command_On_Karaf_Console bundle:list -i | grep ${bundle_name} ${controller} ${karaf_port}
71 BuiltIn.Should_Not_Contain ${output} ${bundle_name}
74 Check_Karaf_Log_Has_Messages
75 [Arguments] ${filter_string} @{message_list}
76 [Documentation] Will succeed if the @{messages} are found in \ the output of "log:display"
77 ${output} = Issue_Command_On_Karaf_Console log:display | grep ${filter_string}
78 : FOR ${message} IN @{message_list}
79 \ BuiltIn.Should_Contain ${output} ${message}
82 Check_Karaf_Log_Message_Count
83 [Arguments] ${message} ${count}
84 [Documentation] Verifies that the ${message} exists in the karaf.log, and checks
85 ... that it appears ${count} number of times.
86 ${output} = Issue_Command_On_Karaf_Console log:display | grep ${message} | wc -l
87 ${line} = String.Get Line ${output} 0
88 ${stripped} = String.Strip String ${line}
89 Should Be Equal As Strings ${stripped} ${count}
92 [Arguments] ${feature_name} ${controller}=${ODL_SYSTEM_IP} ${karaf_port}=${KARAF_SHELL_PORT} ${timeout}=180
93 [Documentation] Will Install the given ${feature_name}
94 BuiltIn.Log ${timeout}
95 ${output} = Issue_Command_On_Karaf_Console feature:install ${feature_name} ${controller} ${karaf_port} ${timeout}
99 Install_A_Feature_Using_Active_Connection
100 [Arguments] ${feature_name}
101 [Documentation] Will Install the given ${feature_name} using active connection
102 ${cmd} = BuiltIn.Set_Variable feature:install ${feature_name}
103 SSHLibrary.Write ${cmd}
104 ${output} SSHLibrary.Read_Until_Regexp ${KARAF_PROMPT}
105 BuiltIn.Log ${output}
109 [Arguments] ${feature_name} ${controller}=${ODL_SYSTEM_IP} ${karaf_port}=${KARAF_SHELL_PORT} ${timeout}=180
110 [Documentation] Will UnInstall the given ${feature_name}
111 ${output} = Issue_Command_On_Karaf_Console feature:uninstall ${feature_name} ${controller} ${karaf_port} ${timeout}
112 BuiltIn.Log ${output}
115 Open_Controller_Karaf_Console_On_Background
116 [Arguments] ${member_index}=${1} ${timeout}=10 ${loglevel}=INFO
117 [Documentation] If there is a stored ssh connection index of connection to the controller's karaf console for ${member_index},
118 ... close the previous connection. In any case create a new connection
119 ... to karaf console for ${member_index}, set correct prompt set and login to karaf console.
120 ... Store connection index for ${member_index} and restore the previous active connection.
121 ${current_ssh_connection_object}= SSHLibrary.Get_Connection
122 BuiltIn.Log ${connection_index_dict}
123 BuiltIn.Log ${member_index}
124 ${status} ${old_connection_index} = BuiltIn.Run_Keyword_And_Ignore_Error Get From Dictionary ${connection_index_dict} ${member_index}
125 BuiltIn.Run_Keyword_If '${status}'=='PASS' BuiltIn.Run_Keywords SSHLibrary.Switch_Connection ${old_connection_index}
126 ... AND SSHLibrary.Close_Connection
127 ${odl_ip} = ClusterManagement.Resolve_IP_Address_For_Member ${member_index}
128 SSHLibrary.Open_Connection ${odl_ip} port=${KARAF_SHELL_PORT} prompt=${KARAF_PROMPT_LOGIN} timeout=${timeout}
129 ${karaf_connection_object} = SSHLibrary.Get_Connection
130 Collections.Set_To_Dictionary ${connection_index_dict} ${member_index} ${karaf_connection_object.index}
131 SSHLibrary.Login ${KARAF_USER} ${KARAF_PASSWORD} loglevel=${loglevel}
132 [Teardown] SSHKeywords.Restore_Current_Ssh_Connection_From_Index ${current_ssh_connection_object.index}
134 Open_Controller_Karaf_Console_With_Timeout
135 [Arguments] ${member_index}=${1} ${timeout}=3s
136 [Documentation] Open new connection to karaf console for member index with specified timeout.
137 BuiltIn.Log ${member_index}
138 ${odl_ip} = ClusterManagement.Resolve_IP_Address_For_Member ${member_index}
139 SSHLibrary.Open_Connection ${odl_ip} port=${KARAF_SHELL_PORT} prompt=${KARAF_PROMPT_LOGIN} timeout=${timeout}
140 SSHLibrary.Login ${KARAF_USER} ${KARAF_PASSWORD}
142 Configure_Timeout_For_Karaf_Console
143 [Arguments] ${timeout} ${member_index_list}=${EMPTY}
144 [Documentation] Configure a different timeout for each Karaf console.
145 ${index_list} = ClusterManagement.List_Indices_Or_All given_list=${member_index_list}
146 ${current_connection_object} = SSHLibrary.Get_Connection
147 : FOR ${member_index} IN @{index_list} # usually: 1, 2, 3
148 \ ${karaf_connection_index} = Collections.Get_From_Dictionary ${connection_index_dict} ${member_index}
149 \ SSHLibrary.Switch_Connection ${karaf_connection_index}
150 \ SSHLibrary.Set_Client_Configuration timeout=${timeout}
151 [Teardown] SSHKeywords.Restore_Current_Ssh_Connection_From_Index ${current_connection_object.index}
153 Execute_Controller_Karaf_Command_On_Background
154 [Arguments] ${command} ${member_index}=${1}
155 [Documentation] Send command to karaf without affecting current SSH connection. Read, log and return response.
156 ${karaf_connection_index} = Collections.Get_From_Dictionary ${connection_index_dict} ${member_index}
157 ${current_connection_index} = SSHLibrary.Switch_Connection ${karaf_connection_index}
158 ${status_write} ${message_write} = BuiltIn.Run_Keyword_And_Ignore_Error SSHLibrary.Write ${command}
159 ${status_wait} ${message_wait} = BuiltIn.Run_Keyword_And_Ignore_Error SSHLibrary.Read_Until_Regexp ${KARAF_PROMPT}
160 BuiltIn.Run Keyword If '${status_write}' != 'PASS' BuiltIn.Fail Failed to send the command: ${command}
161 BuiltIn.Log ${message_wait}
162 BuiltIn.Run_Keyword_If '${status_wait}' != 'PASS' BuiltIn.Fail Failed to see prompt after sending the command: ${command}
163 [Teardown] SSHKeywords.Restore_Current_Ssh_Connection_From_Index ${current_connection_index}
164 [Return] ${message_wait}
166 Execute_Controller_Karaf_Command_With_Retry_On_Background
167 [Arguments] ${command} ${member_index}=${1}
168 [Documentation] Attemp to send command to karaf for ${member_index}, if fail then open connection and try again.
169 ${status} ${message} = BuiltIn.Run_Keyword_And_Ignore_Error Execute_Controller_Karaf_Command_On_Background ${command} ${member_index}
170 BuiltIn.Return_From_Keyword_If '${status}' == 'PASS' ${message}
171 # TODO: Verify this does not leak connections indices.
172 Open_Controller_Karaf_Console_On_Background ${member_index}
173 ${message} = Execute_Controller_Karaf_Command_On_Background ${command} ${member_index}
176 Log_Message_To_Controller_Karaf
177 [Arguments] ${message} ${member_index_list}=${EMPTY} ${tolerate_failure}=True
178 [Documentation] Make sure this resource is initialized. Send a message into the controller's karaf log file on every node listed (or all).
179 ... By default, failure while processing a node is silently ignored, unless ${tolerate_failure} is False.
180 ${index_list} = ClusterManagement.List_Indices_Or_All given_list=${member_index_list}
181 : FOR ${index} IN @{index_list} # usually: 1, 2, 3.
182 \ ${status} ${output} = BuiltIn.Run_Keyword_And_Ignore_Error Execute_Controller_Karaf_Command_With_Retry_On_Background log:log "ROBOT MESSAGE: ${message}" member_index=${index}
183 \ BuiltIn.Run_Keyword_Unless ${tolerate_failure} or "${status}" == "PASS" BuiltIn.Fail ${output}
185 Log_Test_Suite_Start_To_Controller_Karaf
186 [Arguments] ${member_index_list}=${EMPTY}
187 [Documentation] Log suite name to karaf log, useful in suite setup.
188 Log_Message_To_Controller_Karaf Starting suite ${SUITE_SOURCE} ${member_index_list}
190 Log_Testcase_Start_To_Controller_Karaf
191 [Arguments] ${member_index_list}=${EMPTY}
192 [Documentation] Log test case name to karaf log, useful in test case setup.
193 Log_Message_To_Controller_Karaf Starting test ${SUITE_NAME}.${TEST_NAME} ${member_index_list}
195 Set_Bgpcep_Log_Levels
196 [Arguments] ${bgpcep_level}=${DEFAULT_BGPCEP_LOG_LEVEL} ${protocol_level}=${DEFAULT_PROTOCOL_LOG_LEVEL} ${member_index_list}=${EMPTY}
197 [Documentation] Assuming OCKCOB was used, set logging level on bgpcep and protocol loggers without affecting current SSH session.
198 # FIXME: Move to appropriate Resource
199 ${index_list} = ClusterManagement.List_Indices_Or_All given_list=${member_index_list}
200 : FOR ${index} IN @{index_list} # usually: 1, 2, 3.
201 \ Execute_Controller_Karaf_Command_On_Background log:set ${bgpcep_level} org.opendaylight.bgpcep member_index=${index}
202 \ Execute_Controller_Karaf_Command_On_Background log:set ${protocol_level} org.opendaylight.protocol member_index=${index}
204 Get Karaf Log Type From Test Start
205 [Arguments] ${ip} ${test_name} ${type} ${user}=${ODL_SYSTEM_USER} ${password}=${ODL_SYSTEM_PASSWORD} ${prompt}=${ODL_SYSTEM_PROMPT}
206 ... ${log_file}=${KARAF_LOG}
207 [Documentation] Scrapes all log messages that match regexp ${type} which fall after a point given by a log message that
208 ... contains ${test_name}. This is useful if your test cases are marking karaf.log with a message indicating when
209 ... that test case has started; such that you can easily pull out any extra log messsages to parse/log/etc in the
210 ... test logic itself. For example, you can grab all ERRORS that occur during your test case.
211 ${cmd} Set Variable sed '1,/ROBOT MESSAGE: Starting test ${test_name}/d' ${log_file} | grep '${type}'
212 ${output} Run Command On Controller ${ip} ${cmd} ${user} ${password} ${prompt}
215 Get Karaf Log Types From Test Start
216 [Arguments] ${ip} ${test_name} ${types} ${user}=${ODL_SYSTEM_USER} ${password}=${ODL_SYSTEM_PASSWORD} ${prompt}=${ODL_SYSTEM_PROMPT}
217 ... ${log_file}=${KARAF_LOG}
218 [Documentation] A wrapper keyword for "Get Karaf Log Type From Test Start" so that we can parse for multiple types
219 ... of log messages. For example, we can grab all messages of type WARN and ERROR
220 : FOR ${type} IN @{types}
221 \ Get Karaf Log Type From Test Start ${ip} ${test_name} ${type} ${user} ${password}
222 \ ... ${prompt} ${log_file}
224 Get Karaf Log Events From Test Start
225 [Arguments] ${test_name} ${user}=${ODL_SYSTEM_USER} ${password}=${ODL_SYSTEM_PASSWORD} ${prompt}=${ODL_SYSTEM_PROMPT}
226 [Documentation] Wrapper for the wrapper "Get Karaf Log Types From Test Start" so that we can easily loop over
227 ... any number of controllers to analyze karaf.log for ERROR, WARN and Exception log messages
228 ${log_types} = Create List ERROR WARN Exception
229 : FOR ${i} IN RANGE 1 ${NUM_ODL_SYSTEM} + 1
230 \ Get Karaf Log Types From Test Start ${ODL_SYSTEM_${i}_IP} ${test_name} ${log_types}
232 Fail If Exceptions Found During Test
233 [Arguments] ${test_name} ${exceptions_white_list}=${EMPTY}
234 [Documentation] Create a failure if an Exception is found in the karaf.log. Will work for single controller jobs
235 ... as well as 3node cluster jobs
236 : FOR ${i} IN RANGE 1 ${NUM_ODL_SYSTEM} + 1
237 \ Verify Exception Logging In Controller ${ODL_SYSTEM_${i}_IP} ${test_name} ${exceptions_white_list}
239 Verify Exception Logging In Controller
240 [Arguments] ${controller_ip} ${test_name} ${exceptions_white_list}
241 [Documentation] Local keyword to make it easier to loop through N controllers to pull Exceptions from the
242 ... karaf.log file and validate with "Check Against White List"
243 ${exceptions}= Get Karaf Log Type From Test Start ${controller_ip} ${test_name} Exception
244 @{log_lines}= Split String ${exceptions} ${\n}
245 ${num_log_entries} Get Length ${log_lines}
246 Return From Keyword If ${num_log_entries} == ${0} No Exceptions found.
247 : FOR ${log_message} IN @{log_lines}
248 \ Check Against White List ${log_message} ${exceptions_white_list}
250 Check Against White List
251 [Arguments] ${exception_line} ${exceptions_white_list}
252 [Documentation] As soon as the ${exceptions_line} is found in one of the elements of ${exceptions_white_list}
253 ... this keyword will exit and give a Pass to the caller. If there is no match, this keyword will end up
254 ... marking a failure. In the case that no exceptions are found, the caller could end up passing a single
255 ... empty line as that is what is returned when a grep on karaf.log has no match, so we can safely return
256 ... in that case as well.
257 Return From Keyword If "${exception_line}" == ""
258 : FOR ${exception} IN @{exceptions_white_list}
259 \ Return From Keyword If "${exception}" in "${exception_line}" Exceptions found, but whitelisted: ${\n}${exception_line}${\n}
260 Fail Exceptions Found: ${\n}${exception_line}${\n}
263 [Arguments] ${message} ${timeout}=60 ${member_index}=${1}
264 [Documentation] Read karaf logs until message appear
265 # TODO: refactor this keyword to use the new workflow to account for multiple controllers. Initial work was done
266 # in this patch https://git.opendaylight.org/gerrit/#/c/45596/
267 # however, the consumers of this keyword were breaking after that change. Initial theory is that a previous
268 # keyword used before this "Wait For Karaf Log" keyword was closing the karaf console connection, so the
269 # "Flexible SSH Login" keyword from the patch above (45596) was failing.
270 BuiltIn.Log Waiting for '${message}' in karaf log
271 SSHLibrary.Open_Connection ${ODL_SYSTEM_IP} port=${KARAF_SHELL_PORT} prompt=${KARAF_PROMPT_LOGIN} timeout=${timeout}
272 SSHLibrary.Login ${KARAF_USER} ${KARAF_PASSWORD} loglevel=${loglevel}
273 SSHLibrary.Write log:tail
274 SSHLibrary.Read_Until ${message}
275 SSHLibrary.Close_Connection
278 [Arguments] ${bundle_id}
279 [Documentation] Restarts bundle passed as argument. Note this operation is only for testing and not production environments
280 # TODO: prepare this for cluster environment and multiple controllers
281 Execute_Controller_Karaf_Command_With_Retry_On_Background bundle:restart -f $(bundle:id '${bundle_id}')
284 [Documentation] Restarts Karaf and polls log to detect when Karaf is up and running again
285 # TODO: prepare this for cluster environment and multiple controllers
286 Execute_Controller_Karaf_Command_With_Retry_On_Background log:clear
287 Execute_Controller_Karaf_Command_With_Retry_On_Background shutdown -r -f
288 BuiltIn.Run_Keyword_And_Return_Status BuiltIn.Wait_Until_Keyword_Succeeds 240s 60s Wait_For_Karaf_Log Karaf started in
291 [Documentation] Restarts jetty bundle (to reload certificates or key/truststore information)
292 Execute_Controller_Karaf_Command_With_Retry_On_Background log:clear
293 Restart_Bundle OPS4J Pax Web - Jetty
294 Wait_For_Karaf_Log Instantiated the Application class org.opendaylight.restconf.RestconfApplication
295 Wait_For_Karaf_Log Instantiated the Application class org.opendaylight.netconf.sal.rest.impl.RestconfApplication
296 Wait_For_Karaf_Log Instantiated the Application class org.opendaylight.aaa.idm.IdmLightApplication