[Arguments] ${http_timeout}=${DEFAULT_TIMEOUT_HTTP}
[Documentation] Initialize ClusterManagement. Open ssh karaf connections to each ODL.
ClusterManagement.ClusterManagement_Setup http_timeout=${http_timeout}
+ Return From Keyword If '${IS_KARAF_APPL}' == 'False' Not A Karaf App
ClusterManagement.Run_Bash_Command_On_List_Or_All iptables -I INPUT -p tcp --dport ${KARAF_SHELL_PORT} -j ACCEPT; iptables-save
BuiltIn.Comment First connections to Karaf console may fail, so WUKS is used. TODO: Track as a Bug.
FOR ${index} IN @{ClusterManagement__member_index_list}
Verify_Feature_Is_Installed
[Arguments] ${feature_name} ${controller}=${ODL_SYSTEM_IP} ${karaf_port}=${KARAF_SHELL_PORT}
[Documentation] Will Succeed if the given ${feature_name} is found in the output of "feature:list -i"
+ Return From Keyword If '${IS_KARAF_APPL}' == 'False' Not A Karaf App
${output} = Issue_Command_On_Karaf_Console feature:list -i | grep ${feature_name} ${controller} ${karaf_port}
BuiltIn.Should_Contain ${output} ${feature_name}
[Return] ${output}
[Arguments] ${cmd} ${controller}=${ODL_SYSTEM_IP} ${karaf_port}=${KARAF_SHELL_PORT} ${timeout}=10 ${loglevel}=INFO
[Documentation] Will execute the given ${cmd} by ssh'ing to the karaf console running on ${controller}
... Note that this keyword will open&close new SSH connection, without switching back to previously current session.
+ Return From Keyword If '${IS_KARAF_APPL}' == 'False' Not A Karaf App
SSHLibrary.Open_Connection ${controller} port=${karaf_port} prompt=${KARAF_PROMPT_LOGIN} timeout=${timeout}
SSHLibrary.Login ${KARAF_USER} ${KARAF_PASSWORD} loglevel=${loglevel}
SSHLibrary.Write ${cmd}
${output} SSHLibrary.Read_Until_Regexp ${KARAF_PROMPT}
- SSHLibrary.Write logout
+ SSHLibrary.Write_Bare logout\n
SSHLibrary.Close_Connection
BuiltIn.Log ${output}
[Return] ${output}
Safe_Issue_Command_On_Karaf_Console
[Arguments] ${cmd} ${controller}=${ODL_SYSTEM_IP} ${karaf_port}=${KARAF_SHELL_PORT} ${timeout}=10 ${loglevel}=INFO
[Documentation] Run Issue_Command_On_Karaf_Console but restore previous connection afterwards.
+ Return From Keyword If '${IS_KARAF_APPL}' == 'False' Not A Karaf App
BuiltIn.Run_Keyword_And_Return SSHKeywords.Run_Keyword_Preserve_Connection Issue_Command_On_Karaf_Console ${cmd} ${controller} ${karaf_port} ${timeout}
... ${loglevel}
[Arguments] ${cmd} ${elements} ${controller}=${ODL_SYSTEM_IP} ${karaf_port}=${KARAF_SHELL_PORT} ${timeout}=5
[Documentation] Will execute the command using Issue Command On Karaf Console then check for the given elements
... in the command output message
+ Return From Keyword If '${IS_KARAF_APPL}' == 'False' Not A Karaf App
${output} Issue_Command_On_Karaf_Console ${cmd} ${controller} ${karaf_port} ${timeout}
FOR ${i} IN @{elements}
BuiltIn.Should_Contain ${output} ${i}
Verify_Bundle_Is_Installed
[Arguments] ${bundle_name} ${controller}=${ODL_SYSTEM_IP} ${karaf_port}=${KARAF_SHELL_PORT}
[Documentation] Will succeed if the given ${bundle name} is present in the output of "bundle:list -s "
+ Return From Keyword If '${IS_KARAF_APPL}' == 'False' Not A Karaf App
${output} = Issue_Command_On_Karaf_Console bundle:list -s | grep ${bundle_name} ${controller} ${karaf_port}
BuiltIn.Should_Contain ${output} ${bundle_name}
[Return] ${output}
Verify_Bundle_Is_Not_Installed
[Arguments] ${bundle_name} ${controller}=${ODL_SYSTEM_IP} ${karaf_port}=${KARAF_SHELL_PORT}
[Documentation] Will succeed if the given ${bundle_name} is NOT found in the output of "bundle:list -s"
+ Return From Keyword If '${IS_KARAF_APPL}' == 'False' Not A Karaf App
${output} = Issue_Command_On_Karaf_Console bundle:list -i | grep ${bundle_name} ${controller} ${karaf_port}
BuiltIn.Should_Not_Contain ${output} ${bundle_name}
[Return] ${output}
Check_Karaf_Log_Has_Messages
[Arguments] ${filter_string} @{message_list}
[Documentation] Will succeed if the @{messages} are found in \ the output of "log:display"
+ Return From Keyword If '${IS_KARAF_APPL}' == 'False' Not A Karaf App
${output} = Issue_Command_On_Karaf_Console log:display | grep ${filter_string}
FOR ${message} IN @{message_list}
BuiltIn.Should_Contain ${output} ${message}
[Arguments] ${message} ${count} ${use_console}=False
[Documentation] Verifies that the ${message} exists specified number of times in
... karaf console log or Karaf Log Folder based on the arg ${use_console}.
+ Return From Keyword If '${IS_KARAF_APPL}' == 'False' Not A Karaf App
Run Keyword If ${use_console} == False Check_Karaf_Log_File ${message} ${count}
... ELSE Check_Karaf_Log_From_Console ${message} ${count}
[Arguments] ${message} ${count}
[Documentation] Verifies that the ${message} exists in the Karaf Console log:display and checks
... that it appears ${count} number of times
+ Return From Keyword If '${IS_KARAF_APPL}' == 'False' Not A Karaf App
${output} = Issue_Command_On_Karaf_Console log:display | grep ${message} | wc -l
${line} = Get Line ${output} 0
${stripped} = Strip String ${line}
[Arguments] ${message} ${count}
[Documentation] Verifies that the ${message} exists in the Karaf Log Folder and checks
... that it appears ${count} number of times
+ Return From Keyword If '${IS_KARAF_APPL}' == 'False' Not A Karaf App
${output} Run Command On Controller ${ODL_SYSTEM_IP} grep -o ${message} ${WORKSPACE}/${BUNDLEFOLDER}/data/log/* | wc -l
Should Be Equal As Strings ${output} ${count}
Install_A_Feature
[Arguments] ${feature_name} ${controller}=${ODL_SYSTEM_IP} ${karaf_port}=${KARAF_SHELL_PORT} ${timeout}=180
[Documentation] Will Install the given ${feature_name}
+ Return From Keyword If '${IS_KARAF_APPL}' == 'False' Not A Karaf App
BuiltIn.Log ${timeout}
${output} = Issue_Command_On_Karaf_Console feature:install ${feature_name} ${controller} ${karaf_port} ${timeout}
BuiltIn.Log ${output}
Install_A_Feature_Using_Active_Connection
[Arguments] ${feature_name}
[Documentation] Will Install the given ${feature_name} using active connection
+ Return From Keyword If '${IS_KARAF_APPL}' == 'False' Not A Karaf App
${cmd} = BuiltIn.Set_Variable feature:install ${feature_name}
SSHLibrary.Write ${cmd}
${output} SSHLibrary.Read_Until_Regexp ${KARAF_PROMPT}
Uninstall_A_Feature
[Arguments] ${feature_name} ${controller}=${ODL_SYSTEM_IP} ${karaf_port}=${KARAF_SHELL_PORT} ${timeout}=180
[Documentation] Will UnInstall the given ${feature_name}
+ Return From Keyword If '${IS_KARAF_APPL}' == 'False' Not A Karaf App
${output} = Issue_Command_On_Karaf_Console feature:uninstall ${feature_name} ${controller} ${karaf_port} ${timeout}
BuiltIn.Log ${output}
[Return] ${output}
... close the previous connection. In any case create a new connection
... to karaf console for ${member_index}, set correct prompt set and login to karaf console.
... Store connection index for ${member_index} and restore the previous active connection.
+ Return From Keyword If '${IS_KARAF_APPL}' == 'False' Not A Karaf App
${current_ssh_connection_object}= SSHLibrary.Get_Connection
BuiltIn.Log ${connection_index_dict}
BuiltIn.Log ${member_index}
${karaf_connection_object} = SSHLibrary.Get_Connection
Collections.Set_To_Dictionary ${connection_index_dict} ${member_index} ${karaf_connection_object.index}
SSHLibrary.Login ${KARAF_USER} ${KARAF_PASSWORD} loglevel=${loglevel}
- [Teardown] SSHKeywords.Restore_Current_Ssh_Connection_From_Index ${current_ssh_connection_object.index}
+ [Teardown] Run Keyword If '${IS_KARAF_APPL}' == 'True' SSHKeywords.Restore_Current_Ssh_Connection_From_Index ${current_ssh_connection_object.index}
Open_Controller_Karaf_Console_With_Timeout
[Arguments] ${member_index}=${1} ${timeout}=3s
[Documentation] Open new connection to karaf console for member index with specified timeout.
+ Return From Keyword If '${IS_KARAF_APPL}' == 'False' Not A Karaf App
BuiltIn.Log ${member_index}
${odl_ip} = ClusterManagement.Resolve_IP_Address_For_Member ${member_index}
SSHLibrary.Open_Connection ${odl_ip} port=${KARAF_SHELL_PORT} prompt=${KARAF_PROMPT_LOGIN} timeout=${timeout}
Configure_Timeout_For_Karaf_Console
[Arguments] ${timeout} ${member_index_list}=${EMPTY}
[Documentation] Configure a different timeout for each Karaf console.
+ Return From Keyword If '${IS_KARAF_APPL}' == 'False' Not A Karaf App
${index_list} = ClusterManagement.List_Indices_Or_All given_list=${member_index_list}
${current_connection_object} = SSHLibrary.Get_Connection
FOR ${member_index} IN @{index_list} # usually: 1, 2, 3
SSHLibrary.Switch_Connection ${karaf_connection_index}
SSHLibrary.Set_Client_Configuration timeout=${timeout}
END
- [Teardown] SSHKeywords.Restore_Current_Ssh_Connection_From_Index ${current_connection_object.index}
+ [Teardown] Run Keyword If '${IS_KARAF_APPL}' == 'True' SSHKeywords.Restore_Current_Ssh_Connection_From_Index ${current_connection_object.index}
Execute_Controller_Karaf_Command_On_Background
[Arguments] ${command} ${member_index}=${1}
[Documentation] Send command to karaf without affecting current SSH connection. Read, log and return response.
+ Return From Keyword If '${IS_KARAF_APPL}' == 'False' Not A Karaf App
${karaf_connection_index} = Collections.Get_From_Dictionary ${connection_index_dict} ${member_index}
${current_connection_index} = SSHLibrary.Switch_Connection ${karaf_connection_index}
${status_write} ${message_write} = BuiltIn.Run_Keyword_And_Ignore_Error SSHLibrary.Write ${command}
BuiltIn.Run Keyword If '${status_write}' != 'PASS' BuiltIn.Fail Failed to send the command: ${command}
BuiltIn.Log ${message_wait}
BuiltIn.Run_Keyword_If '${status_wait}' != 'PASS' BuiltIn.Fail Failed to see prompt after sending the command: ${command}
- [Teardown] SSHKeywords.Restore_Current_Ssh_Connection_From_Index ${current_connection_index}
+ [Teardown] Run Keyword If '${IS_KARAF_APPL}' == 'True' SSHKeywords.Restore_Current_Ssh_Connection_From_Index ${current_connection_index}
[Return] ${message_wait}
Execute_Controller_Karaf_Command_With_Retry_On_Background
[Arguments] ${command} ${member_index}=${1}
[Documentation] Attemp to send command to karaf for ${member_index}, if fail then open connection and try again.
+ Return From Keyword If '${IS_KARAF_APPL}' == 'False' Not A Karaf App
${status} ${message} = BuiltIn.Run_Keyword_And_Ignore_Error Execute_Controller_Karaf_Command_On_Background ${command} ${member_index}
BuiltIn.Return_From_Keyword_If '${status}' == 'PASS' ${message}
# TODO: Verify this does not leak connections indices.
[Arguments] ${message} ${member_index_list}=${EMPTY} ${tolerate_failure}=True
[Documentation] Make sure this resource is initialized. Send a message into the controller's karaf log file on every node listed (or all).
... By default, failure while processing a node is silently ignored, unless ${tolerate_failure} is False.
+ Return From Keyword If '${IS_KARAF_APPL}' == 'False' Not A Karaf App
${index_list} = ClusterManagement.List_Indices_Or_All given_list=${member_index_list}
FOR ${index} IN @{index_list} # usually: 1, 2, 3.
${status} ${output} = BuiltIn.Run_Keyword_And_Ignore_Error Execute_Controller_Karaf_Command_With_Retry_On_Background log:log "ROBOT MESSAGE: ${message}" member_index=${index}
Log_Test_Suite_Start_To_Controller_Karaf
[Arguments] ${member_index_list}=${EMPTY}
[Documentation] Log suite name to karaf log, useful in suite setup.
+ Return From Keyword If '${IS_KARAF_APPL}' == 'False' Not A Karaf App
Log_Message_To_Controller_Karaf Starting suite ${SUITE_SOURCE} ${member_index_list}
Log_Testcase_Start_To_Controller_Karaf
[Arguments] ${member_index_list}=${EMPTY}
[Documentation] Log test case name to karaf log, useful in test case setup.
+ Return From Keyword If '${IS_KARAF_APPL}' == 'False' Not A Karaf App
Log_Message_To_Controller_Karaf Starting test ${SUITE_NAME}.${TEST_NAME} ${member_index_list}
Set_Bgpcep_Log_Levels
[Arguments] ${bgpcep_level}=${DEFAULT_BGPCEP_LOG_LEVEL} ${protocol_level}=${DEFAULT_PROTOCOL_LOG_LEVEL} ${member_index_list}=${EMPTY}
[Documentation] Assuming OCKCOB was used, set logging level on bgpcep and protocol loggers without affecting current SSH session.
# FIXME: Move to appropriate Resource
+ Return From Keyword If '${IS_KARAF_APPL}' == 'False' Not A Karaf App
${index_list} = ClusterManagement.List_Indices_Or_All given_list=${member_index_list}
FOR ${index} IN @{index_list} # usually: 1, 2, 3.
Execute_Controller_Karaf_Command_On_Background log:set ${bgpcep_level} org.opendaylight.bgpcep member_index=${index}
... contains ${test_name}. This is useful if your test cases are marking karaf.log with a message indicating when
... that test case has started; such that you can easily pull out any extra log messsages to parse/log/etc in the
... test logic itself. For example, you can grab all ERRORS that occur during your test case.
+ Return From Keyword If '${IS_KARAF_APPL}' == 'False' Not A Karaf App
${output} = Run Command On Controller ${ip} ${cmd} ${user} ${password} ${prompt}
@{log_lines} = Split String ${output} ${\n}
[Return] ${log_lines}
[Arguments] ${test_name} ${log_file}=${KARAF_LOG} ${fail}=False
[Documentation] Create a failure if an Exception is found in the karaf.log that has not been whitelisted.
... Will work for single controller jobs as well as 3node cluster jobs
+ Return From Keyword If '${IS_KARAF_APPL}' == 'False' Not A Karaf App
FOR ${i} IN RANGE 1 ${NUM_ODL_SYSTEM} + 1
${cmd} = Set Variable sed '1,/ROBOT MESSAGE: Starting test ${test_name}/d' ${log_file}
${output} = Get Karaf Log Lines From Test Start ${ODL_SYSTEM_${i}_IP} ${test_name} ${cmd}
Log And Fail Exceptions
[Arguments] ${exlist} ${listlength}
[Documentation] Print the list of failed exceptions and fail the test
+ Return From Keyword If '${IS_KARAF_APPL}' == 'False' Not A Karaf App
Collections.Log List ${exlist}
${exstr} = BuiltIn.Catenate ${exlist}
BuiltIn.Fail New exceptions found: ${listlength}\n${exstr}
... contains ${test_name}. This is useful if your test cases are marking karaf.log with a message indicating when
... that test case has started; such that you can easily pull out any extra log messsages to parse/log/etc in the
... test logic itself. For example, you can grab all ERRORS that occur during your test case.
+ Return From Keyword If '${IS_KARAF_APPL}' == 'False' Not A Karaf App
${cmd} Set Variable sed '1,/ROBOT MESSAGE: Starting test ${test_name}/d' ${log_file} | grep '${type}'
${output} Run Command On Controller ${ip} ${cmd} ${user} ${password} ${prompt}
[Return] ${output}
... ${log_file}=${KARAF_LOG}
[Documentation] A wrapper keyword for "Get Karaf Log Type From Test Start" so that we can parse for multiple types
... of log messages. For example, we can grab all messages of type WARN and ERROR
+ Return From Keyword If '${IS_KARAF_APPL}' == 'False' Not A Karaf App
FOR ${type} IN @{types}
Get Karaf Log Type From Test Start ${ip} ${test_name} ${type} ${user} ${password}
... ${prompt} ${log_file}
[Arguments] ${test_name} ${user}=${ODL_SYSTEM_USER} ${password}=${ODL_SYSTEM_PASSWORD} ${prompt}=${ODL_SYSTEM_PROMPT}
[Documentation] Wrapper for the wrapper "Get Karaf Log Types From Test Start" so that we can easily loop over
... any number of controllers to analyze karaf.log for ERROR, WARN and Exception log messages
+ Return From Keyword If '${IS_KARAF_APPL}' == 'False' Not A Karaf App
${log_types} = Create List ERROR WARN Exception
FOR ${i} IN RANGE 1 ${NUM_ODL_SYSTEM} + 1
Get Karaf Log Types From Test Start ${ODL_SYSTEM_${i}_IP} ${test_name} ${log_types}
[Arguments] ${test_name} ${exceptions_white_list}=${EMPTY}
[Documentation] Create a failure if an Exception is found in the karaf.log. Will work for single controller jobs
... as well as 3node cluster jobs
+ Return From Keyword If '${IS_KARAF_APPL}' == 'False' Not A Karaf App
FOR ${i} IN RANGE 1 ${NUM_ODL_SYSTEM} + 1
Verify Exception Logging In Controller ${ODL_SYSTEM_${i}_IP} ${test_name} ${exceptions_white_list}
END
[Arguments] ${controller_ip} ${test_name} ${exceptions_white_list}
[Documentation] Local keyword to make it easier to loop through N controllers to pull Exceptions from the
... karaf.log file and validate with "Check Against White List"
+ Return From Keyword If '${IS_KARAF_APPL}' == 'False' Not A Karaf App
${exceptions}= Get Karaf Log Type From Test Start ${controller_ip} ${test_name} Exception
@{log_lines}= Split String ${exceptions} ${\n}
${num_log_entries} Get Length ${log_lines}
... marking a failure. In the case that no exceptions are found, the caller could end up passing a single
... empty line as that is what is returned when a grep on karaf.log has no match, so we can safely return
... in that case as well.
+ Return From Keyword If '${IS_KARAF_APPL}' == 'False' Not A Karaf App
Return From Keyword If "${exception_line}" == ""
FOR ${exception} IN @{exceptions_white_list}
Return From Keyword If "${exception}" in "${exception_line}" Exceptions found, but whitelisted: ${\n}${exception_line}${\n}
# however, the consumers of this keyword were breaking after that change. Initial theory is that a previous
# keyword used before this "Wait For Karaf Log" keyword was closing the karaf console connection, so the
# "Flexible SSH Login" keyword from the patch above (45596) was failing.
+ Return From Keyword If '${IS_KARAF_APPL}' == 'False' Not A Karaf App
BuiltIn.Log Waiting for '${message}' in karaf log
SSHLibrary.Open_Connection ${ODL_SYSTEM_IP} port=${KARAF_SHELL_PORT} prompt=${KARAF_PROMPT_LOGIN} timeout=${timeout}
SSHLibrary.Login ${KARAF_USER} ${KARAF_PASSWORD} loglevel=${loglevel}
[Arguments] ${bundle_id}
[Documentation] Restarts bundle passed as argument. Note this operation is only for testing and not production environments
# TODO: prepare this for cluster environment and multiple controllers
+ Return From Keyword If '${IS_KARAF_APPL}' == 'False' Not A Karaf App
Execute_Controller_Karaf_Command_With_Retry_On_Background bundle:restart $(bundle:id '${bundle_id}')
Restart_Karaf
[Documentation] Restarts Karaf and polls log to detect when Karaf is up and running again
# TODO: prepare this for cluster environment and multiple controllers
+ Return From Keyword If '${IS_KARAF_APPL}' == 'False' Not A Karaf App
Execute_Controller_Karaf_Command_With_Retry_On_Background log:clear
Execute_Controller_Karaf_Command_With_Retry_On_Background shutdown -r -f
BuiltIn.Run_Keyword_And_Return_Status BuiltIn.Wait_Until_Keyword_Succeeds 240s 60s Wait_For_Karaf_Log Shiro environment initialized in
Restart_Jetty
[Documentation] Restarts jetty bundle (to reload certificates or key/truststore information)
+ Return From Keyword If '${IS_KARAF_APPL}' == 'False' Not A Karaf App
Execute_Controller_Karaf_Command_With_Retry_On_Background log:clear
Restart_Bundle OPS4J Pax Web - Jetty
Wait_For_Karaf_Log Started jetty-default
Code Review / integration / test.git / blobdiff