X-Git-Url: https://git.opendaylight.org/gerrit/gitweb?a=blobdiff_plain;f=csit%2Flibraries%2FKarafKeywords.robot;h=4670711cf073e64ac215fd1911e4dabfa62e344a;hb=9fd9a4e44c0bffe67832bb61a406813041f19bb8;hp=e9260d0e844c754ae75f87450176e7801c302f1c;hpb=87c6de79a247b8b5d4dd047863bcd3dfd93f9199;p=integration%2Ftest.git diff --git a/csit/libraries/KarafKeywords.robot b/csit/libraries/KarafKeywords.robot index e9260d0e84..4670711cf0 100644 --- a/csit/libraries/KarafKeywords.robot +++ b/csit/libraries/KarafKeywords.robot @@ -1,9 +1,13 @@ *** Settings *** -Documentation Karaf library. This library is useful to deal with controller Karaf console for ssh sessions in cluster. +Documentation Karaf library. General utility keywords for interacting with the karaf environment, such as the +... karaf console, karaf.log, karaf features, and karaf config files. +... +... This library is useful to deal with controller Karaf console for ssh sessions in cluster. ... Running Setup_Karaf_Keywords is necessary. If SetupUtils initialization is called, this gets initialized as well. ... If this gets initialized, ClusterManagement gets initialized as well. Library SSHLibrary Library OperatingSystem +Library ${CURDIR}/netvirt/excepts.py Resource ${CURDIR}/ClusterManagement.robot Resource ${CURDIR}/SSHKeywords.robot Variables ${CURDIR}/../variables/Variables.py @@ -14,9 +18,10 @@ ${connection_index_dict} &{EMPTY} *** Keywords *** Setup_Karaf_Keywords - [Arguments] ${http_timeout}=1 + [Arguments] ${http_timeout}=${DEFAULT_TIMEOUT_HTTP} [Documentation] Initialize ClusterManagement. Open ssh karaf connections to each ODL. ClusterManagement.ClusterManagement_Setup http_timeout=${http_timeout} + ClusterManagement.Run_Bash_Command_On_List_Or_All iptables -I INPUT -p tcp --dport ${KARAF_SHELL_PORT} -j ACCEPT; iptables-save BuiltIn.Comment First connections to Karaf console may fail, so WUKS is used. TODO: Track as a Bug. : FOR ${index} IN @{ClusterManagement__member_index_list} \ BuiltIn.Run_Keyword_And_Ignore_Error BuiltIn.Wait_Until_Keyword_Succeeds 3s 1s Open_Controller_Karaf_Console_On_Background member_index=${index} @@ -32,10 +37,11 @@ Issue_Command_On_Karaf_Console [Arguments] ${cmd} ${controller}=${ODL_SYSTEM_IP} ${karaf_port}=${KARAF_SHELL_PORT} ${timeout}=10 ${loglevel}=INFO [Documentation] Will execute the given ${cmd} by ssh'ing to the karaf console running on ${controller} ... Note that this keyword will open&close new SSH connection, without switching back to previously current session. - SSHLibrary.Open_Connection ${controller} port=${karaf_port} prompt=${KARAF_PROMPT} timeout=${timeout} + SSHLibrary.Open_Connection ${controller} port=${karaf_port} prompt=${KARAF_PROMPT_LOGIN} timeout=${timeout} SSHLibrary.Login ${KARAF_USER} ${KARAF_PASSWORD} loglevel=${loglevel} SSHLibrary.Write ${cmd} - ${output} SSHLibrary.Read_Until ${KARAF_PROMPT} + ${output} SSHLibrary.Read_Until_Regexp ${KARAF_PROMPT} + SSHLibrary.Write logout SSHLibrary.Close_Connection BuiltIn.Log ${output} [Return] ${output} @@ -76,6 +82,29 @@ Check_Karaf_Log_Has_Messages \ BuiltIn.Should_Contain ${output} ${message} [Return] ${output} +Check_Karaf_Log_Message_Count + [Arguments] ${message} ${count} ${use_console}=False + [Documentation] Verifies that the ${message} exists specified number of times in + ... karaf console log or Karaf Log Folder based on the arg ${use_console}. + Run Keyword If ${use_console} == False Check_Karaf_Log_File ${message} ${count} + ... ELSE Check_Karaf_Log_From_Console ${message} ${count} + +Check_Karaf_Log_From_Console + [Arguments] ${message} ${count} + [Documentation] Verifies that the ${message} exists in the Karaf Console log:display and checks + ... that it appears ${count} number of times + ${output} = Issue_Command_On_Karaf_Console log:display | grep ${message} | wc -l + ${line} = Get Line ${output} 0 + ${stripped} = Strip String ${line} + Should Be Equal As Strings ${stripped} ${count} + +Check_Karaf_Log_File + [Arguments] ${message} ${count} + [Documentation] Verifies that the ${message} exists in the Karaf Log Folder and checks + ... that it appears ${count} number of times + ${output} Run Command On Controller ${ODL_SYSTEM_IP} grep -o ${message} ${WORKSPACE}/${BUNDLEFOLDER}/data/log/* | wc -l + Should Be Equal As Strings ${output} ${count} + Install_A_Feature [Arguments] ${feature_name} ${controller}=${ODL_SYSTEM_IP} ${karaf_port}=${KARAF_SHELL_PORT} ${timeout}=180 [Documentation] Will Install the given ${feature_name} @@ -89,7 +118,7 @@ Install_A_Feature_Using_Active_Connection [Documentation] Will Install the given ${feature_name} using active connection ${cmd} = BuiltIn.Set_Variable feature:install ${feature_name} SSHLibrary.Write ${cmd} - ${output} SSHLibrary.Read_Until ${KARAF_PROMPT} + ${output} SSHLibrary.Read_Until_Regexp ${KARAF_PROMPT} BuiltIn.Log ${output} [Return] ${output} @@ -111,9 +140,10 @@ Open_Controller_Karaf_Console_On_Background BuiltIn.Log ${member_index} ${status} ${old_connection_index} = BuiltIn.Run_Keyword_And_Ignore_Error Get From Dictionary ${connection_index_dict} ${member_index} BuiltIn.Run_Keyword_If '${status}'=='PASS' BuiltIn.Run_Keywords SSHLibrary.Switch_Connection ${old_connection_index} + ... AND SSHLibrary.Write logout ... AND SSHLibrary.Close_Connection ${odl_ip} = ClusterManagement.Resolve_IP_Address_For_Member ${member_index} - SSHLibrary.Open_Connection ${odl_ip} port=${KARAF_SHELL_PORT} prompt=${KARAF_DETAILED_PROMPT} timeout=${timeout} + SSHLibrary.Open_Connection ${odl_ip} port=${KARAF_SHELL_PORT} prompt=${KARAF_PROMPT_LOGIN} timeout=${timeout} ${karaf_connection_object} = SSHLibrary.Get_Connection Collections.Set_To_Dictionary ${connection_index_dict} ${member_index} ${karaf_connection_object.index} SSHLibrary.Login ${KARAF_USER} ${KARAF_PASSWORD} loglevel=${loglevel} @@ -124,7 +154,7 @@ Open_Controller_Karaf_Console_With_Timeout [Documentation] Open new connection to karaf console for member index with specified timeout. BuiltIn.Log ${member_index} ${odl_ip} = ClusterManagement.Resolve_IP_Address_For_Member ${member_index} - SSHLibrary.Open_Connection ${odl_ip} port=${KARAF_SHELL_PORT} prompt=${KARAF_DETAILED_PROMPT} timeout=${timeout} + SSHLibrary.Open_Connection ${odl_ip} port=${KARAF_SHELL_PORT} prompt=${KARAF_PROMPT_LOGIN} timeout=${timeout} SSHLibrary.Login ${KARAF_USER} ${KARAF_PASSWORD} Configure_Timeout_For_Karaf_Console @@ -144,7 +174,7 @@ Execute_Controller_Karaf_Command_On_Background ${karaf_connection_index} = Collections.Get_From_Dictionary ${connection_index_dict} ${member_index} ${current_connection_index} = SSHLibrary.Switch_Connection ${karaf_connection_index} ${status_write} ${message_write} = BuiltIn.Run_Keyword_And_Ignore_Error SSHLibrary.Write ${command} - ${status_wait} ${message_wait} = BuiltIn.Run_Keyword_And_Ignore_Error SSHLibrary.Read_Until_Prompt + ${status_wait} ${message_wait} = BuiltIn.Run_Keyword_And_Ignore_Error SSHLibrary.Read_Until_Regexp ${KARAF_PROMPT} BuiltIn.Run Keyword If '${status_write}' != 'PASS' BuiltIn.Fail Failed to send the command: ${command} BuiltIn.Log ${message_wait} BuiltIn.Run_Keyword_If '${status_wait}' != 'PASS' BuiltIn.Fail Failed to see prompt after sending the command: ${command} @@ -178,7 +208,7 @@ Log_Test_Suite_Start_To_Controller_Karaf Log_Testcase_Start_To_Controller_Karaf [Arguments] ${member_index_list}=${EMPTY} [Documentation] Log test case name to karaf log, useful in test case setup. - Log_Message_To_Controller_Karaf Starting test ${TEST_NAME} ${member_index_list} + Log_Message_To_Controller_Karaf Starting test ${SUITE_NAME}.${TEST_NAME} ${member_index_list} Set_Bgpcep_Log_Levels [Arguments] ${bgpcep_level}=${DEFAULT_BGPCEP_LOG_LEVEL} ${protocol_level}=${DEFAULT_PROTOCOL_LOG_LEVEL} ${member_index_list}=${EMPTY} @@ -189,6 +219,95 @@ Set_Bgpcep_Log_Levels \ Execute_Controller_Karaf_Command_On_Background log:set ${bgpcep_level} org.opendaylight.bgpcep member_index=${index} \ Execute_Controller_Karaf_Command_On_Background log:set ${protocol_level} org.opendaylight.protocol member_index=${index} +Get Karaf Log Lines From Test Start + [Arguments] ${ip} ${test_name} ${cmd} ${user}=${ODL_SYSTEM_USER} ${password}=${ODL_SYSTEM_PASSWORD} ${prompt}=${ODL_SYSTEM_PROMPT} + ... ${log_file}=${KARAF_LOG} + [Documentation] Scrapes all log messages that match regexp ${type} which fall after a point given by a log message that + ... contains ${test_name}. This is useful if your test cases are marking karaf.log with a message indicating when + ... that test case has started; such that you can easily pull out any extra log messsages to parse/log/etc in the + ... test logic itself. For example, you can grab all ERRORS that occur during your test case. + ${output} = Run Command On Controller ${ip} ${cmd} ${user} ${password} ${prompt} + @{log_lines} = Split String ${output} ${\n} + [Return] ${log_lines} + +Fail If Exceptions Found During Test + [Arguments] ${test_name} ${log_file}=${KARAF_LOG} + [Documentation] Create a failure if an Exception is found in the karaf.log that has not been whitelisted. + ... Will work for single controller jobs as well as 3node cluster jobs + : FOR ${i} IN RANGE 1 ${NUM_ODL_SYSTEM} + 1 + \ ${cmd} = Set Variable sed '1,/ROBOT MESSAGE: Starting test ${test_name}/d' ${log_file} + \ ${output} = Get Karaf Log Lines From Test Start ${ODL_SYSTEM_${i}_IP} ${test_name} ${cmd} + \ ${exlist} ${matchlist} = Verify Exceptions ${output} + \ Write Exceptions Map To File ${SUITE_NAME}.${TEST_NAME} /tmp/odl${i}_exceptions.txt + \ ${listlength} = BuiltIn.Get Length ${exlist} + \ BuiltIn.Run Keyword If ${listlength} != 0 Log And Fail Exceptions ${exlist} ${listlength} + \ ... ELSE Collections.Log List ${matchlist} + +Log And Fail Exceptions + [Arguments] ${exlist} ${listlength} + [Documentation] Print the list of failed exceptions and fail the test + Collections.Log List ${exlist} + ${exstr} = BuiltIn.Catenate ${exlist} + BuiltIn.Fail New exceptions found: ${listlength}\n${exstr} + +Get Karaf Log Type From Test Start + [Arguments] ${ip} ${test_name} ${type} ${user}=${ODL_SYSTEM_USER} ${password}=${ODL_SYSTEM_PASSWORD} ${prompt}=${ODL_SYSTEM_PROMPT} + ... ${log_file}=${KARAF_LOG} + [Documentation] Scrapes all log messages that match regexp ${type} which fall after a point given by a log message that + ... contains ${test_name}. This is useful if your test cases are marking karaf.log with a message indicating when + ... that test case has started; such that you can easily pull out any extra log messsages to parse/log/etc in the + ... test logic itself. For example, you can grab all ERRORS that occur during your test case. + ${cmd} Set Variable sed '1,/ROBOT MESSAGE: Starting test ${test_name}/d' ${log_file} | grep '${type}' + ${output} Run Command On Controller ${ip} ${cmd} ${user} ${password} ${prompt} + [Return] ${output} + +Get Karaf Log Types From Test Start + [Arguments] ${ip} ${test_name} ${types} ${user}=${ODL_SYSTEM_USER} ${password}=${ODL_SYSTEM_PASSWORD} ${prompt}=${ODL_SYSTEM_PROMPT} + ... ${log_file}=${KARAF_LOG} + [Documentation] A wrapper keyword for "Get Karaf Log Type From Test Start" so that we can parse for multiple types + ... of log messages. For example, we can grab all messages of type WARN and ERROR + : FOR ${type} IN @{types} + \ Get Karaf Log Type From Test Start ${ip} ${test_name} ${type} ${user} ${password} + \ ... ${prompt} ${log_file} + +Get Karaf Log Events From Test Start + [Arguments] ${test_name} ${user}=${ODL_SYSTEM_USER} ${password}=${ODL_SYSTEM_PASSWORD} ${prompt}=${ODL_SYSTEM_PROMPT} + [Documentation] Wrapper for the wrapper "Get Karaf Log Types From Test Start" so that we can easily loop over + ... any number of controllers to analyze karaf.log for ERROR, WARN and Exception log messages + ${log_types} = Create List ERROR WARN Exception + : FOR ${i} IN RANGE 1 ${NUM_ODL_SYSTEM} + 1 + \ Get Karaf Log Types From Test Start ${ODL_SYSTEM_${i}_IP} ${test_name} ${log_types} + +Fail If Exceptions Found During Test Deprecated + [Arguments] ${test_name} ${exceptions_white_list}=${EMPTY} + [Documentation] Create a failure if an Exception is found in the karaf.log. Will work for single controller jobs + ... as well as 3node cluster jobs + : FOR ${i} IN RANGE 1 ${NUM_ODL_SYSTEM} + 1 + \ Verify Exception Logging In Controller ${ODL_SYSTEM_${i}_IP} ${test_name} ${exceptions_white_list} + +Verify Exception Logging In Controller + [Arguments] ${controller_ip} ${test_name} ${exceptions_white_list} + [Documentation] Local keyword to make it easier to loop through N controllers to pull Exceptions from the + ... karaf.log file and validate with "Check Against White List" + ${exceptions}= Get Karaf Log Type From Test Start ${controller_ip} ${test_name} Exception + @{log_lines}= Split String ${exceptions} ${\n} + ${num_log_entries} Get Length ${log_lines} + Return From Keyword If ${num_log_entries} == ${0} No Exceptions found. + : FOR ${log_message} IN @{log_lines} + \ Check Against White List ${log_message} ${exceptions_white_list} + +Check Against White List + [Arguments] ${exception_line} ${exceptions_white_list} + [Documentation] As soon as the ${exceptions_line} is found in one of the elements of ${exceptions_white_list} + ... this keyword will exit and give a Pass to the caller. If there is no match, this keyword will end up + ... marking a failure. In the case that no exceptions are found, the caller could end up passing a single + ... empty line as that is what is returned when a grep on karaf.log has no match, so we can safely return + ... in that case as well. + Return From Keyword If "${exception_line}" == "" + : FOR ${exception} IN @{exceptions_white_list} + \ Return From Keyword If "${exception}" in "${exception_line}" Exceptions found, but whitelisted: ${\n}${exception_line}${\n} + Fail Exceptions Found: ${\n}${exception_line}${\n} + Wait_For_Karaf_Log [Arguments] ${message} ${timeout}=60 ${member_index}=${1} [Documentation] Read karaf logs until message appear @@ -198,28 +317,29 @@ Wait_For_Karaf_Log # keyword used before this "Wait For Karaf Log" keyword was closing the karaf console connection, so the # "Flexible SSH Login" keyword from the patch above (45596) was failing. BuiltIn.Log Waiting for '${message}' in karaf log - SSHLibrary.Open_Connection ${ODL_SYSTEM_IP} port=${KARAF_SHELL_PORT} prompt=${KARAF_PROMPT} timeout=${timeout} - BuiltIn.Login ${KARAF_USER} ${KARAF_PASSWORD} loglevel=${loglevel} + SSHLibrary.Open_Connection ${ODL_SYSTEM_IP} port=${KARAF_SHELL_PORT} prompt=${KARAF_PROMPT_LOGIN} timeout=${timeout} + SSHLibrary.Login ${KARAF_USER} ${KARAF_PASSWORD} loglevel=${loglevel} SSHLibrary.Write log:tail SSHLibrary.Read_Until ${message} + SSHLibrary.Write logout SSHLibrary.Close_Connection Restart_Bundle [Arguments] ${bundle_id} [Documentation] Restarts bundle passed as argument. Note this operation is only for testing and not production environments # TODO: prepare this for cluster environment and multiple controllers - Safe_Issue_Command_On_Karaf_Console bundle:restart -f $(bundle:id '${bundle_id}') + Execute_Controller_Karaf_Command_With_Retry_On_Background bundle:restart -f $(bundle:id '${bundle_id}') Restart_Karaf [Documentation] Restarts Karaf and polls log to detect when Karaf is up and running again # TODO: prepare this for cluster environment and multiple controllers - Safe_Issue_Command_On_Karaf_Console log:clear - Issue_Command_On_Karaf_Console shutdown -r -f + Execute_Controller_Karaf_Command_With_Retry_On_Background log:clear + Execute_Controller_Karaf_Command_With_Retry_On_Background shutdown -r -f BuiltIn.Run_Keyword_And_Return_Status BuiltIn.Wait_Until_Keyword_Succeeds 240s 60s Wait_For_Karaf_Log Karaf started in Restart_Jetty [Documentation] Restarts jetty bundle (to reload certificates or key/truststore information) - Safe_Issue_Command_On_Karaf_Console log:clear + Execute_Controller_Karaf_Command_With_Retry_On_Background log:clear Restart_Bundle OPS4J Pax Web - Jetty Wait_For_Karaf_Log Instantiated the Application class org.opendaylight.restconf.RestconfApplication Wait_For_Karaf_Log Instantiated the Application class org.opendaylight.netconf.sal.rest.impl.RestconfApplication