X-Git-Url: https://git.opendaylight.org/gerrit/gitweb?a=blobdiff_plain;f=csit%2Flibraries%2FKarafKeywords.robot;h=3ccea296e2a7fbf1a18bd70e3fbb35892e45cea3;hb=f7292aa70b3bb674423383f92003405424beb71b;hp=9b58d5d32f769fc3ef8eca847f87ba89428c2ab8;hpb=b7213f5f60b3301e2aca9587c8466f87bde30d8d;p=integration%2Ftest.git diff --git a/csit/libraries/KarafKeywords.robot b/csit/libraries/KarafKeywords.robot index 9b58d5d32f..3ccea296e2 100644 --- a/csit/libraries/KarafKeywords.robot +++ b/csit/libraries/KarafKeywords.robot @@ -1,57 +1,67 @@ *** Settings *** -Documentation Karaf library. This library is useful to deal with controller Karaf console. +Documentation Karaf library. This library is useful to deal with controller Karaf console for ssh sessions in cluster. +... Running Setup Karaf Keywords is necessary. If SetupUtils initialization is called, this gets initialized as well. +... If this gets initialized, ClusterManagement gets initialized as well. Library SSHLibrary Library OperatingSystem -Variables ../variables/Variables.py +Resource ${CURDIR}/ClusterManagement.robot +Resource ${CURDIR}/SSHKeywords.robot +Variables ${CURDIR}/../variables/Variables.py *** Variables *** ${WORKSPACE} /tmp -${BUNDLEFOLDER} distribution-karaf-0.3.0-SNAPSHOT -${KarafKeywords__karaf_connection_index} -1 +${connection_index_dict} &{EMPTY} *** Keywords *** -Check Karaf Log File Does Not Have Messages - [Arguments] ${ip} ${message} ${user}=${CONTROLLER_USER} ${password}=${CONTROLLER_PASSWORD} ${prompt}=${DEFAULT_LINUX_PROMPT} ${log_file}=${WORKSPACE}/${BUNDLEFOLDER}/data/log/karaf.log - [Documentation] Fails if the provided ${message} is found in the karaf.log file. Uses grep to search. The - ... karaf.log file can be overridden with ${log_file} to be any file on the given system @ ${ip} - ${output}= Run Command On Controller ${ip} grep ${message} ${log_file} user=${user} password=${password} prompt=${prompt} - Should Not Contain ${output} ${message} +Setup Karaf Keywords + [Documentation] Initialize ClusterManagement. Open ssh karaf connections to each ODL. + ClusterManagement.ClusterManagement_Setup + BuiltIn.Comment First connections to Karaf console may fail, so WUKS is used. TODO: Track as a Bug. + : FOR ${index} IN @{ClusterManagement__member_index_list} + \ BuiltIn.Wait_Until_Keyword_Succeeds 10x 0.2s Open Controller Karaf Console On Background member_index=${index} Verify Feature Is Installed - [Arguments] ${feature_name} ${controller}=${CONTROLLER} ${karaf_port}=${KARAF_SHELL_PORT} + [Arguments] ${feature_name} ${controller}=${ODL_SYSTEM_IP} ${karaf_port}=${KARAF_SHELL_PORT} [Documentation] Will Succeed if the given ${feature_name} is found in the output of "feature:list -i" ${output}= Issue Command On Karaf Console feature:list -i | grep ${feature_name} ${controller} ${karaf_port} Should Contain ${output} ${feature_name} [Return] ${output} -Verify Feature Is Not Installed - [Arguments] ${feature_name} ${controller}=${CONTROLLER} ${karaf_port}=${KARAF_SHELL_PORT} - [Documentation] Will Succeed if the given ${feature_name} is NOT found in the output of "feature:list -i" - ${output}= Issue Command On Karaf Console feature:list -i | grep ${feature_name} ${controller} ${karaf_port} - Should Not Contain ${output} ${feature_name} - [Return] ${output} - Issue Command On Karaf Console - [Arguments] ${cmd} ${controller}=${CONTROLLER} ${karaf_port}=${KARAF_SHELL_PORT} ${timeout}=5 - [Documentation] Will execute the given ${cmd} by ssh'ing to the karaf console running on ${CONTROLLER} + [Arguments] ${cmd} ${controller}=${ODL_SYSTEM_IP} ${karaf_port}=${KARAF_SHELL_PORT} ${timeout}=10 ${loglevel}=INFO + [Documentation] Will execute the given ${cmd} by ssh'ing to the karaf console running on ${controller} ... Note that this keyword will open&close new SSH connection, without switching back to previously current session. Open Connection ${controller} port=${karaf_port} prompt=${KARAF_PROMPT} timeout=${timeout} - Login ${KARAF_USER} ${KARAF_PASSWORD} + Login ${KARAF_USER} ${KARAF_PASSWORD} loglevel=${loglevel} Write ${cmd} ${output} Read Until ${KARAF_PROMPT} Close Connection Log ${output} [Return] ${output} +Safe_Issue_Command_On_Karaf_Console + [Arguments] ${cmd} ${controller}=${ODL_SYSTEM_IP} ${karaf_port}=${KARAF_SHELL_PORT} ${timeout}=10 ${loglevel}=INFO + [Documentation] Run Issue_Command_On_Karaf_Console but restore previous connection afterwards. + BuiltIn.Run_Keyword_And_Return SSHKeywords.Run_Keyword_Preserve_Connection Issue_Command_On_Karaf_Console ${cmd} ${controller} ${karaf_port} ${timeout} + ... ${loglevel} + +Check For Elements On Karaf Command Output Message + [Arguments] ${cmd} ${elements} ${controller}=${ODL_SYSTEM_IP} ${karaf_port}=${KARAF_SHELL_PORT} ${timeout}=5 + [Documentation] Will execute the command using Issue Command On Karaf Console then check for the given elements + ... in the command output message + ${output} Issue Command On Karaf Console ${cmd} ${controller} ${karaf_port} ${timeout} + : FOR ${i} IN @{elements} + \ Should Contain ${output} ${i} + Verify Bundle Is Installed - [Arguments] ${bundle_name} ${controller}=${CONTROLLER} ${karaf_port}=${KARAF_SHELL_PORT} + [Arguments] ${bundle_name} ${controller}=${ODL_SYSTEM_IP} ${karaf_port}=${KARAF_SHELL_PORT} [Documentation] Will succeed if the given ${bundle name} is present in the output of "bundle:list -s " ${output}= Issue Command On Karaf Console bundle:list -s | grep ${bundle_name} ${controller} ${karaf_port} Should Contain ${output} ${bundle_name} [Return] ${output} Verify Bundle Is Not Installed - [Arguments] ${bundle_name} ${controller}=${CONTROLLER} ${karaf_port}=${KARAF_SHELL_PORT} + [Arguments] ${bundle_name} ${controller}=${ODL_SYSTEM_IP} ${karaf_port}=${KARAF_SHELL_PORT} [Documentation] Will succeed if the given ${bundle_name} is NOT found in the output of "bundle:list -s" ${output}= Issue Command On Karaf Console bundle:list -i | grep ${bundle_name} ${controller} ${karaf_port} Should Not Contain ${output} ${bundle_name} @@ -66,95 +76,146 @@ Check Karaf Log Has Messages [Return] ${output} Install a Feature - [Arguments] ${feature_name} ${controller}=${CONTROLLER} ${karaf_port}=${KARAF_SHELL_PORT} ${timeout}=15 + [Arguments] ${feature_name} ${controller}=${ODL_SYSTEM_IP} ${karaf_port}=${KARAF_SHELL_PORT} ${timeout}=180 [Documentation] Will Install the given ${feature_name} Log ${timeout} ${output}= Issue Command On Karaf Console feature:install ${feature_name} ${controller} ${karaf_port} ${timeout} Log ${output} [Return] ${output} +Install a Feature Using Active Connection + [Arguments] ${feature_name} + [Documentation] Will Install the given ${feature_name} using active connection + ${cmd}= BuiltIn.Set_Variable feature:install ${feature_name} + Write ${cmd} + ${output} SSHLibrary.Read_Until ${KARAF_PROMPT} + Log ${output} + [Return] ${output} + Uninstall a Feature - [Arguments] ${feature_name} ${controller}=${CONTROLLER} ${karaf_port}=${KARAF_SHELL_PORT} ${timeout}=15 + [Arguments] ${feature_name} ${controller}=${ODL_SYSTEM_IP} ${karaf_port}=${KARAF_SHELL_PORT} ${timeout}=180 [Documentation] Will UnInstall the given ${feature_name} ${output}= Issue Command On Karaf Console feature:uninstall ${feature_name} ${controller} ${karaf_port} ${timeout} Log ${output} [Return] ${output} -Restore Current SSH Connection From Index - [Arguments] ${connection_index} - [Documentation] Restore active SSH connection in SSHLibrary to given index. - ... - ... Restore the currently active connection state in - ... SSHLibrary to match the state returned by "Switch - ... Connection" or "Get Connection". More specifically makes - ... sure that there will be no active connection when the - ... \${connection_index} reported by these means is None. - ... - ... There is a misfeature in SSHLibrary: Invoking "SSHLibrary.Switch_Connection" - ... and passing None as the "index_or_alias" argument to it has exactly the - ... same effect as invoking "Close Connection". - ... https://github.com/robotframework/SSHLibrary/blob/master/src/SSHLibrary/library.py#L560 - ... - ... We want to have Keyword which will "switch out" to previous - ... "no connection active" state without killing the background one. - ... - ... As some suites may hypothetically rely on non-writability of active connection, - ... workaround is applied by opening and closing temporary connection. - ... Unfortunately this will fail if run on Jython and there is no SSH server - ... running on localhost, port 22 but there is nothing easy that can be done about it. - BuiltIn.Run Keyword And Return If ${connection_index} is not None SSHLibrary.Switch Connection ${connection_index} - # The background connection is still current, bury it. - SSHLibrary.Open Connection 127.0.0.1 - SSHLibrary.Close Connection - Open Controller Karaf Console On Background - [Documentation] Connect to the controller's karaf console, but do not switch to it. - ${current_ssh_connection}= SSHLibrary.Get Connection - SSHLibrary.Open Connection ${CONTROLLER} port=${KARAF_SHELL_PORT} prompt=${KARAF_DETAILED_PROMPT} - ${karaf_connection}= SSHLibrary.Get Connection + [Arguments] ${member_index}=${1} ${timeout}=10 ${loglevel}=INFO + [Documentation] If there is a stored ssh connection index of connection to the controller's karaf console for ${member_index}, + ... close the previous connection. In any case create a new connection + ... to karaf console for ${member_index}, set correct prompt set and login to karaf console. + ... Store connection index for ${member_index} and restore the previous active connection. + ${current_ssh_connection_object}= SSHLibrary.Get Connection + BuiltIn.Log ${connection_index_dict} + BuiltIn.Log ${member_index} + ${status} ${old_connection_index} = BuiltIn.Run Keyword And Ignore Error Get From Dictionary ${connection_index_dict} ${member_index} + BuiltIn.Run Keyword If '${status}'=='PASS' BuiltIn.Run Keywords SSHLibrary.Switch Connection ${old_connection_index} + ... AND SSHLibrary.Close Connection + ${odl_ip}= ClusterManagement.Resolve_IP_Address_For_Member ${member_index} + SSHLibrary.Open Connection ${odl_ip} port=${KARAF_SHELL_PORT} prompt=${KARAF_DETAILED_PROMPT} timeout=${timeout} + ${karaf_connection_object}= SSHLibrary.Get Connection + Collections.Set To Dictionary ${connection_index_dict} ${member_index} ${karaf_connection_object.index} + SSHLibrary.Login ${KARAF_USER} ${KARAF_PASSWORD} loglevel=${loglevel} + [Teardown] SSHKeywords.Restore Current SSH Connection From Index ${current_ssh_connection_object.index} + +Open Controller Karaf Console With Timeout + [Arguments] ${member_index}=${1} ${timeout}=3s + [Documentation] Open new connection to karaf console for member index with specified timeout. + BuiltIn.Log ${member_index} + ${odl_ip}= ClusterManagement.Resolve_IP_Address_For_Member ${member_index} + SSHLibrary.Open Connection ${odl_ip} port=${KARAF_SHELL_PORT} prompt=${KARAF_DETAILED_PROMPT} timeout=${timeout} SSHLibrary.Login ${KARAF_USER} ${KARAF_PASSWORD} - BuiltIn.Set Suite Variable ${KarafKeywords__karaf_connection_index} ${karaf_connection.index} - Restore Current SSH Connection From Index ${current_ssh_connection.index} Configure Timeout For Karaf Console - [Arguments] ${timeout} - [Documentation] Configure a different timeout for the Karaf console - BuiltIn.Run Keyword If ${KarafKeywords__karaf_connection_index} == -1 Fail Need to connect to a Karaf Console first - ${current_connection_index}= SSHLibrary.Switch Connection ${KarafKeywords__karaf_connection_index} - SSHLibrary.Set_Client_Configuration timeout=${timeout} - Restore Current SSH Connection From Index ${current_connection_index} + [Arguments] ${timeout} ${member_index_list}=${EMPTY} + [Documentation] Configure a different timeout for each Karaf console. + ${index_list} = ClusterManagement.List Indices Or All given_list=${member_index_list} + ${current_connection_object}= SSHLibrary.Get Connection + : FOR ${member_index} IN @{index_list} # usually: 1, 2, 3 + \ ${karaf_connection_index}= Collections.Get From Dictionary ${connection_index_dict} ${member_index} + \ SSHLibrary.Switch Connection ${karaf_connection_index} + \ SSHLibrary.Set_Client_Configuration timeout=${timeout} + [Teardown] SSHKeywords.Restore Current SSH Connection From Index ${current_connection_object.index} Execute Controller Karaf Command On Background - [Arguments] ${command} + [Arguments] ${command} ${member_index}=${1} [Documentation] Send command to karaf without affecting current SSH connection. Read, log and return response. - ... This assumes Karaf connection has index saved and correct prompt set. - BuiltIn.Run Keyword If ${KarafKeywords__karaf_connection_index} == -1 Fail Need to connect to a Karaf Console first - ${current_connection_index}= SSHLibrary.Switch Connection ${KarafKeywords__karaf_connection_index} + ${karaf_connection_index}= Collections.Get From Dictionary ${connection_index_dict} ${member_index} + ${current_connection_index}= SSHLibrary.Switch Connection ${karaf_connection_index} ${status_write} ${message_write}= BuiltIn.Run Keyword And Ignore Error SSHLibrary.Write ${command} ${status_wait} ${message_wait}= BuiltIn.Run Keyword And Ignore Error SSHLibrary.Read Until Prompt - Restore Current SSH Connection From Index ${current_connection_index} BuiltIn.Run Keyword If '${status_write}' != 'PASS' BuiltIn.Fail Failed to send the command: ${command} BuiltIn.Log ${message_wait} BuiltIn.Run Keyword If '${status_wait}' != 'PASS' BuiltIn.Fail Failed to see prompt after sending the command: ${command} + [Teardown] SSHKeywords.Restore Current SSH Connection From Index ${current_connection_index} [Return] ${message_wait} +Execute Controller Karaf Command With Retry On Background + [Arguments] ${command} ${member_index}=${1} + [Documentation] Attemp to send command to karaf for ${member_index}, if fail then open connection and try again. + ${status} ${message}= BuiltIn.Run Keyword And Ignore Error Execute Controller Karaf Command On Background ${command} ${member_index} + BuiltIn.Return_From_Keyword_If '${status}' == 'PASS' ${message} + # TODO: Verify this does not leak connections indices. + Open Controller Karaf Console On Background ${member_index} + ${message}= Execute Controller Karaf Command On Background ${command} ${member_index} + [Return] ${message} + Log Message To Controller Karaf - [Arguments] ${message} - [Documentation] Send a message into the controller's karaf log file. Do not change current SSH connection. - ${reply}= Execute Controller Karaf Command On Background log:log "ROBOT MESSAGE: ${message}" - [Return] ${reply} + [Arguments] ${message} ${member_index_list}=${EMPTY} ${tolerate_failure}=True + [Documentation] Make sure this resource is initialized. Send a message into the controller's karaf log file on every node listed (or all). + ... By default, failure while processing a node is silently ignored, unless ${tolerate_failure} is False. + ${index_list} = ClusterManagement.List Indices Or All given_list=${member_index_list} + : FOR ${index} IN @{index_list} # usually: 1, 2, 3. + \ ${status} ${output}= BuiltIn.Run Keyword And Ignore Error Execute Controller Karaf Command With Retry On Background log:log "ROBOT MESSAGE: ${message}" member_index=${index} + \ BuiltIn.Run_Keyword_Unless ${tolerate_failure} or "${status}" == "PASS" BuiltIn.Fail ${output} Log Test Suite Start To Controller Karaf + [Arguments] ${member_index_list}=${EMPTY} [Documentation] Log suite name to karaf log, useful in suite setup. - Log Message To Controller Karaf Starting suite ${SUITE_SOURCE} + Log Message To Controller Karaf Starting suite ${SUITE_SOURCE} ${member_index_list} Log Testcase Start To Controller Karaf + [Arguments] ${member_index_list}=${EMPTY} [Documentation] Log test case name to karaf log, useful in test case setup. - Log Message To Controller Karaf Starting test ${TEST_NAME} + Log Message To Controller Karaf Starting test ${TEST_NAME} ${member_index_list} Set Bgpcep Log Levels - [Arguments] ${bgpcep_level}=${DEFAULT_BGPCEP_LOG_LEVEL} ${protocol_level}=${DEFAULT_PROTOCOL_LOG_LEVEL} + [Arguments] ${bgpcep_level}=${DEFAULT_BGPCEP_LOG_LEVEL} ${protocol_level}=${DEFAULT_PROTOCOL_LOG_LEVEL} ${member_index_list}=${EMPTY} [Documentation] Assuming OCKCOB was used, set logging level on bgpcep and protocol loggers without affecting current SSH session. # FIXME: Move to appropriate Resource - Execute Controller Karaf Command On Background log:set ${bgpcep_level} org.opendaylight.bgpcep - Execute Controller Karaf Command On Background log:set ${protocol_level} org.opendaylight.protocol + ${index_list} = ClusterManagement.List Indices Or All given_list=${member_index_list} + : FOR ${index} IN @{index_list} # usually: 1, 2, 3. + \ Execute Controller Karaf Command On Background log:set ${bgpcep_level} org.opendaylight.bgpcep member_index=${index} + \ Execute Controller Karaf Command On Background log:set ${protocol_level} org.opendaylight.protocol member_index=${index} + +Wait For Karaf Log + [Arguments] ${message} ${timeout}=60 ${member_index}=${1} + [Documentation] Read karaf logs until message appear + # TODO: refactor this keyword to use the new workflow to account for multiple controllers. Initial work was done + # in this patch https://git.opendaylight.org/gerrit/#/c/45596/ + # however, the consumers of this keyword were breaking after that change. Initial theory is that a previous + # keyword used before this "Wait For Karaf Log" keyword was closing the karaf console connection, so the + # "Flexible SSH Login" keyword from the patch above (45596) was failing. + Log Waiting for '${message}' in karaf log + Open Connection ${ODL_SYSTEM_IP} port=${KARAF_SHELL_PORT} prompt=${KARAF_PROMPT} timeout=${timeout} + Login ${KARAF_USER} ${KARAF_PASSWORD} loglevel=${loglevel} + Write log:tail + Read Until ${message} + Close Connection + +Restart Bundle + [Arguments] ${bundle_id} + [Documentation] Restarts bundle passed as argument. Note this operation is only for testing and not production environments + # TODO: prepare this for cluster environment and multiple controllers + Safe_Issue_Command_On_Karaf_Console bundle:restart -f $(bundle:id ${bundle_id}) + +Restart Karaf + [Documentation] Restarts Karaf and polls log to detect when Karaf is up and running again + # TODO: prepare this for cluster environment and multiple controllers + Safe_Issue_Command_On_Karaf_Console log:clear + Issue_Command_On_Karaf_Console shutdown -r -f + Run Keyword And Return Status Wait Until Keyword Succeeds 240s 60s Wait For Karaf Log Karaf started in + +Restart Jetty + [Documentation] Restarts jetty bundle (to reload certificates or key/truststore information) + Restart Bundle OPS4J Pax Web - Jetty