... If this gets initialized, ClusterManagement gets initialized as well.
Library SSHLibrary
Library OperatingSystem
+Library ${CURDIR}/netvirt/excepts.py
Resource ${CURDIR}/ClusterManagement.robot
Resource ${CURDIR}/SSHKeywords.robot
Variables ${CURDIR}/../variables/Variables.py
ClusterManagement.ClusterManagement_Setup http_timeout=${http_timeout}
ClusterManagement.Run_Bash_Command_On_List_Or_All iptables -I INPUT -p tcp --dport ${KARAF_SHELL_PORT} -j ACCEPT; iptables-save
BuiltIn.Comment First connections to Karaf console may fail, so WUKS is used. TODO: Track as a Bug.
- : FOR ${index} IN @{ClusterManagement__member_index_list}
- \ BuiltIn.Run_Keyword_And_Ignore_Error BuiltIn.Wait_Until_Keyword_Succeeds 3s 1s Open_Controller_Karaf_Console_On_Background member_index=${index}
+ FOR ${index} IN @{ClusterManagement__member_index_list}
+ BuiltIn.Run_Keyword_And_Ignore_Error BuiltIn.Wait_Until_Keyword_Succeeds 3s 1s Open_Controller_Karaf_Console_On_Background member_index=${index}
+ END
Verify_Feature_Is_Installed
[Arguments] ${feature_name} ${controller}=${ODL_SYSTEM_IP} ${karaf_port}=${KARAF_SHELL_PORT}
SSHLibrary.Login ${KARAF_USER} ${KARAF_PASSWORD} loglevel=${loglevel}
SSHLibrary.Write ${cmd}
${output} SSHLibrary.Read_Until_Regexp ${KARAF_PROMPT}
+ SSHLibrary.Write logout
SSHLibrary.Close_Connection
BuiltIn.Log ${output}
[Return] ${output}
[Documentation] Will execute the command using Issue Command On Karaf Console then check for the given elements
... in the command output message
${output} Issue_Command_On_Karaf_Console ${cmd} ${controller} ${karaf_port} ${timeout}
- : FOR ${i} IN @{elements}
- \ BuiltIn.Should_Contain ${output} ${i}
+ FOR ${i} IN @{elements}
+ BuiltIn.Should_Contain ${output} ${i}
+ END
Verify_Bundle_Is_Installed
[Arguments] ${bundle_name} ${controller}=${ODL_SYSTEM_IP} ${karaf_port}=${KARAF_SHELL_PORT}
[Arguments] ${filter_string} @{message_list}
[Documentation] Will succeed if the @{messages} are found in \ the output of "log:display"
${output} = Issue_Command_On_Karaf_Console log:display | grep ${filter_string}
- : FOR ${message} IN @{message_list}
- \ BuiltIn.Should_Contain ${output} ${message}
+ FOR ${message} IN @{message_list}
+ BuiltIn.Should_Contain ${output} ${message}
+ END
[Return] ${output}
Check_Karaf_Log_Message_Count
BuiltIn.Log ${member_index}
${status} ${old_connection_index} = BuiltIn.Run_Keyword_And_Ignore_Error Get From Dictionary ${connection_index_dict} ${member_index}
BuiltIn.Run_Keyword_If '${status}'=='PASS' BuiltIn.Run_Keywords SSHLibrary.Switch_Connection ${old_connection_index}
- ... AND SSHLibrary.Close_Connection
+ ... AND BuiltIn.Run_Keyword_And_Ignore_Error SSHLibrary.Write logout
+ ... AND BuiltIn.Run_Keyword_And_Ignore_Error SSHLibrary.Close_Connection
${odl_ip} = ClusterManagement.Resolve_IP_Address_For_Member ${member_index}
SSHLibrary.Open_Connection ${odl_ip} port=${KARAF_SHELL_PORT} prompt=${KARAF_PROMPT_LOGIN} timeout=${timeout}
${karaf_connection_object} = SSHLibrary.Get_Connection
[Documentation] Configure a different timeout for each Karaf console.
${index_list} = ClusterManagement.List_Indices_Or_All given_list=${member_index_list}
${current_connection_object} = SSHLibrary.Get_Connection
- : FOR ${member_index} IN @{index_list} # usually: 1, 2, 3
- \ ${karaf_connection_index} = Collections.Get_From_Dictionary ${connection_index_dict} ${member_index}
- \ SSHLibrary.Switch_Connection ${karaf_connection_index}
- \ SSHLibrary.Set_Client_Configuration timeout=${timeout}
+ FOR ${member_index} IN @{index_list} # usually: 1, 2, 3
+ ${karaf_connection_index} = Collections.Get_From_Dictionary ${connection_index_dict} ${member_index}
+ SSHLibrary.Switch_Connection ${karaf_connection_index}
+ SSHLibrary.Set_Client_Configuration timeout=${timeout}
+ END
[Teardown] SSHKeywords.Restore_Current_Ssh_Connection_From_Index ${current_connection_object.index}
Execute_Controller_Karaf_Command_On_Background
[Documentation] Make sure this resource is initialized. Send a message into the controller's karaf log file on every node listed (or all).
... By default, failure while processing a node is silently ignored, unless ${tolerate_failure} is False.
${index_list} = ClusterManagement.List_Indices_Or_All given_list=${member_index_list}
- : FOR ${index} IN @{index_list} # usually: 1, 2, 3.
- \ ${status} ${output} = BuiltIn.Run_Keyword_And_Ignore_Error Execute_Controller_Karaf_Command_With_Retry_On_Background log:log "ROBOT MESSAGE: ${message}" member_index=${index}
- \ BuiltIn.Run_Keyword_Unless ${tolerate_failure} or "${status}" == "PASS" BuiltIn.Fail ${output}
+ FOR ${index} IN @{index_list} # usually: 1, 2, 3.
+ ${status} ${output} = BuiltIn.Run_Keyword_And_Ignore_Error Execute_Controller_Karaf_Command_With_Retry_On_Background log:log "ROBOT MESSAGE: ${message}" member_index=${index}
+ BuiltIn.Run_Keyword_Unless ${tolerate_failure} or "${status}" == "PASS" BuiltIn.Fail ${output}
+ END
Log_Test_Suite_Start_To_Controller_Karaf
[Arguments] ${member_index_list}=${EMPTY}
[Documentation] Assuming OCKCOB was used, set logging level on bgpcep and protocol loggers without affecting current SSH session.
# FIXME: Move to appropriate Resource
${index_list} = ClusterManagement.List_Indices_Or_All given_list=${member_index_list}
- : FOR ${index} IN @{index_list} # usually: 1, 2, 3.
- \ Execute_Controller_Karaf_Command_On_Background log:set ${bgpcep_level} org.opendaylight.bgpcep member_index=${index}
- \ Execute_Controller_Karaf_Command_On_Background log:set ${protocol_level} org.opendaylight.protocol member_index=${index}
+ FOR ${index} IN @{index_list} # usually: 1, 2, 3.
+ Execute_Controller_Karaf_Command_On_Background log:set ${bgpcep_level} org.opendaylight.bgpcep member_index=${index}
+ Execute_Controller_Karaf_Command_On_Background log:set ${protocol_level} org.opendaylight.protocol member_index=${index}
+ END
+
+Get Karaf Log Lines From Test Start
+ [Arguments] ${ip} ${test_name} ${cmd} ${user}=${ODL_SYSTEM_USER} ${password}=${ODL_SYSTEM_PASSWORD} ${prompt}=${ODL_SYSTEM_PROMPT}
+ ... ${log_file}=${KARAF_LOG}
+ [Documentation] Scrapes all log messages that match regexp ${type} which fall after a point given by a log message that
+ ... contains ${test_name}. This is useful if your test cases are marking karaf.log with a message indicating when
+ ... that test case has started; such that you can easily pull out any extra log messsages to parse/log/etc in the
+ ... test logic itself. For example, you can grab all ERRORS that occur during your test case.
+ ${output} = Run Command On Controller ${ip} ${cmd} ${user} ${password} ${prompt}
+ @{log_lines} = Split String ${output} ${\n}
+ [Return] ${log_lines}
+
+Fail If Exceptions Found During Test
+ [Arguments] ${test_name} ${log_file}=${KARAF_LOG} ${fail}=False
+ [Documentation] Create a failure if an Exception is found in the karaf.log that has not been whitelisted.
+ ... Will work for single controller jobs as well as 3node cluster jobs
+ FOR ${i} IN RANGE 1 ${NUM_ODL_SYSTEM} + 1
+ ${cmd} = Set Variable sed '1,/ROBOT MESSAGE: Starting test ${test_name}/d' ${log_file}
+ ${output} = Get Karaf Log Lines From Test Start ${ODL_SYSTEM_${i}_IP} ${test_name} ${cmd}
+ ${exlist} ${matchlist} = Verify Exceptions ${output}
+ Write Exceptions Map To File ${SUITE_NAME}.${TEST_NAME} /tmp/odl${i}_exceptions.txt
+ ${listlength} = BuiltIn.Get Length ${exlist}
+ BuiltIn.Run Keyword If "${fail}"=="True" and ${listlength} != 0 Log And Fail Exceptions ${exlist} ${listlength}
+ ... ELSE Collections.Log List ${matchlist}
+ END
+
+Log And Fail Exceptions
+ [Arguments] ${exlist} ${listlength}
+ [Documentation] Print the list of failed exceptions and fail the test
+ Collections.Log List ${exlist}
+ ${exstr} = BuiltIn.Catenate ${exlist}
+ BuiltIn.Fail New exceptions found: ${listlength}\n${exstr}
Get Karaf Log Type From Test Start
[Arguments] ${ip} ${test_name} ${type} ${user}=${ODL_SYSTEM_USER} ${password}=${ODL_SYSTEM_PASSWORD} ${prompt}=${ODL_SYSTEM_PROMPT}
... ${log_file}=${KARAF_LOG}
[Documentation] A wrapper keyword for "Get Karaf Log Type From Test Start" so that we can parse for multiple types
... of log messages. For example, we can grab all messages of type WARN and ERROR
- : FOR ${type} IN @{types}
- \ Get Karaf Log Type From Test Start ${ip} ${test_name} ${type} ${user} ${password}
- \ ... ${prompt} ${log_file}
+ FOR ${type} IN @{types}
+ Get Karaf Log Type From Test Start ${ip} ${test_name} ${type} ${user} ${password}
+ ... ${prompt} ${log_file}
+ END
Get Karaf Log Events From Test Start
[Arguments] ${test_name} ${user}=${ODL_SYSTEM_USER} ${password}=${ODL_SYSTEM_PASSWORD} ${prompt}=${ODL_SYSTEM_PROMPT}
[Documentation] Wrapper for the wrapper "Get Karaf Log Types From Test Start" so that we can easily loop over
... any number of controllers to analyze karaf.log for ERROR, WARN and Exception log messages
${log_types} = Create List ERROR WARN Exception
- : FOR ${i} IN RANGE 1 ${NUM_ODL_SYSTEM} + 1
- \ Get Karaf Log Types From Test Start ${ODL_SYSTEM_${i}_IP} ${test_name} ${log_types}
+ FOR ${i} IN RANGE 1 ${NUM_ODL_SYSTEM} + 1
+ Get Karaf Log Types From Test Start ${ODL_SYSTEM_${i}_IP} ${test_name} ${log_types}
+ END
-Fail If Exceptions Found During Test
+Fail If Exceptions Found During Test Deprecated
[Arguments] ${test_name} ${exceptions_white_list}=${EMPTY}
[Documentation] Create a failure if an Exception is found in the karaf.log. Will work for single controller jobs
... as well as 3node cluster jobs
- : FOR ${i} IN RANGE 1 ${NUM_ODL_SYSTEM} + 1
- \ Verify Exception Logging In Controller ${ODL_SYSTEM_${i}_IP} ${test_name} ${exceptions_white_list}
+ FOR ${i} IN RANGE 1 ${NUM_ODL_SYSTEM} + 1
+ Verify Exception Logging In Controller ${ODL_SYSTEM_${i}_IP} ${test_name} ${exceptions_white_list}
+ END
Verify Exception Logging In Controller
[Arguments] ${controller_ip} ${test_name} ${exceptions_white_list}
${exceptions}= Get Karaf Log Type From Test Start ${controller_ip} ${test_name} Exception
@{log_lines}= Split String ${exceptions} ${\n}
${num_log_entries} Get Length ${log_lines}
- # https://jira.opendaylight.org/browse/ODLPARENT-152
- Run Keyword And Return If ${num_log_entries} == ${2} Ignore ODLPARENT-152 @{log_lines}
Return From Keyword If ${num_log_entries} == ${0} No Exceptions found.
- : FOR ${log_message} IN @{log_lines}
- \ Check Against White List ${log_message} ${exceptions_white_list}
+ FOR ${log_message} IN @{log_lines}
+ Check Against White List ${log_message} ${exceptions_white_list}
+ END
Check Against White List
[Arguments] ${exception_line} ${exceptions_white_list}
... empty line as that is what is returned when a grep on karaf.log has no match, so we can safely return
... in that case as well.
Return From Keyword If "${exception_line}" == ""
- : FOR ${exception} IN @{exceptions_white_list}
- \ Return From Keyword If "${exception}" in "${exception_line}" Exceptions found, but whitelisted: ${\n}${exception_line}${\n}
+ FOR ${exception} IN @{exceptions_white_list}
+ Return From Keyword If "${exception}" in "${exception_line}" Exceptions found, but whitelisted: ${\n}${exception_line}${\n}
+ END
Fail Exceptions Found: ${\n}${exception_line}${\n}
-Ignore ODLPARENT-152
- [Arguments] @{log_lines}
- [Documentation] This will ignore exceptions in @{log_lines} if they match the Exception pattern found in the list
- ... variable @{ODLPARENT_152}. Everything else will result in this keyword Failing.
- Return From Keyword If "@{ODLPARENT_152}[0]" in "@{log_lines}[0]" and "@{ODLPARENT_152}[1]" in "@{log_lines}[1]"
- Fail Exceptions found and not in the whitelist to ignore
-
Wait_For_Karaf_Log
[Arguments] ${message} ${timeout}=60 ${member_index}=${1}
[Documentation] Read karaf logs until message appear
SSHLibrary.Login ${KARAF_USER} ${KARAF_PASSWORD} loglevel=${loglevel}
SSHLibrary.Write log:tail
SSHLibrary.Read_Until ${message}
+ SSHLibrary.Write logout
SSHLibrary.Close_Connection
Restart_Bundle
[Arguments] ${bundle_id}
[Documentation] Restarts bundle passed as argument. Note this operation is only for testing and not production environments
# TODO: prepare this for cluster environment and multiple controllers
- Execute_Controller_Karaf_Command_With_Retry_On_Background bundle:restart -f $(bundle:id '${bundle_id}')
+ Execute_Controller_Karaf_Command_With_Retry_On_Background bundle:restart $(bundle:id '${bundle_id}')
Restart_Karaf
[Documentation] Restarts Karaf and polls log to detect when Karaf is up and running again
# TODO: prepare this for cluster environment and multiple controllers
Execute_Controller_Karaf_Command_With_Retry_On_Background log:clear
Execute_Controller_Karaf_Command_With_Retry_On_Background shutdown -r -f
- BuiltIn.Run_Keyword_And_Return_Status BuiltIn.Wait_Until_Keyword_Succeeds 240s 60s Wait_For_Karaf_Log Karaf started in
+ BuiltIn.Run_Keyword_And_Return_Status BuiltIn.Wait_Until_Keyword_Succeeds 240s 60s Wait_For_Karaf_Log Shiro environment initialized in
Restart_Jetty
[Documentation] Restarts jetty bundle (to reload certificates or key/truststore information)
Execute_Controller_Karaf_Command_With_Retry_On_Background log:clear
Restart_Bundle OPS4J Pax Web - Jetty
- Wait_For_Karaf_Log Instantiated the Application class org.opendaylight.restconf.RestconfApplication
- Wait_For_Karaf_Log Instantiated the Application class org.opendaylight.netconf.sal.rest.impl.RestconfApplication
- Wait_For_Karaf_Log Instantiated the Application class org.opendaylight.aaa.idm.IdmLightApplication
+ Wait_For_Karaf_Log Started jetty-default