From: Vratko Polak Date: Thu, 4 May 2017 13:03:38 +0000 (+0200) Subject: Clean data when switching tell-base protocol. X-Git-Tag: release/carbon~109 X-Git-Url: https://git.opendaylight.org/gerrit/gitweb?a=commitdiff_plain;h=b2e5cdbe5ee97bd1be24a26d2e627da96456516c;p=integration%2Ftest.git Clean data when switching tell-base protocol. That is to make sure persisted data does not create failures. That can be reverted when suite results are stable enough. SSHKeywords.Run_Keyword_With_Ssh added to be more defensive against situation when inner failure prevents session restore. ClusterManagement.Check_Bash_Command_On_List_Or_All added to rely on this defensivenes when running bash commands with logging and checks. Change-Id: I9ac004324f06646aec582987cd4665a61db88b4f Signed-off-by: Vratko Polak --- diff --git a/csit/libraries/ClusterManagement.robot b/csit/libraries/ClusterManagement.robot index 473d859057..23c79de01e 100644 --- a/csit/libraries/ClusterManagement.robot +++ b/csit/libraries/ClusterManagement.robot @@ -489,17 +489,39 @@ Flush_Iptables_From_List_Or_All ${command} = BuiltIn.Set_Variable sudo iptables -v -F ${output} = Run_Bash_Command_On_List_Or_All command=${command} member_index_list=${member_index_list} +Check_Bash_Command_On_List_Or_All + [Arguments] ${command} ${member_index_list}=${EMPTY} ${return_success_only}=False ${log_on_success}=True ${log_on_failure}=True ${stderr_must_be_empty}=True + [Documentation] Cycle through indices (or all), run bash command on each, using temporary SSH session and restoring the previously active one. + ${index_list} = List_Indices_Or_All given_list=${member_index_list} + : FOR ${index} IN @{index_list} + \ Check_Bash_Command_On_Member command=${command} member_index=${index} return_success_only=${return_success_only} log_on_success=${log_on_success} log_on_failure=${log_on_failure} + \ ... stderr_must_be_empty=${stderr_must_be_empty} + +Check_Bash_Command_On_Member + [Arguments] ${command} ${member_index} ${return_success_only}=False ${log_on_success}=True ${log_on_failure}=True ${stderr_must_be_empty}=True + [Documentation] Open SSH session, call SSHKeywords.Execute_Command_Passes, close session, restore previously active session and return output. + BuiltIn.Run_Keyword_And_Return SSHKeywords.Run_Keyword_Preserve_Connection Check_Unsafely_Bash_Command_On_Member ${command} ${member_index} return_success_only=${return_success_only} log_on_success=${log_on_success} + ... log_on_failure=${log_on_failure} stderr_must_be_empty=${stderr_must_be_empty} + +Check_Unsafely_Bash_Command_On_Member + [Arguments] ${command} ${member_index} ${return_success_only}=False ${log_on_success}=True ${log_on_failure}=True ${stderr_must_be_empty}=True + [Documentation] Obtain Ip address, open session, call SSHKeywords.Execute_Command_Passes, close session and return output. This affects which SSH session is active. + ${member_ip} = Resolve_Ip_Address_For_Member ${member_index} + BuiltIn.Run_Keyword_And_Return SSHKeywords.Run_Unsafely_Keyword_Over_Temporary_Odl_Session ${member_ip} Execute_Command_Passes ${command} return_success_only=${return_success_only} log_on_success=${log_on_success} + ... log_on_failure=${log_on_failure} stderr_must_be_empty=${stderr_must_be_empty} + Run_Bash_Command_On_List_Or_All [Arguments] ${command} ${member_index_list}=${EMPTY} [Documentation] Cycle through indices (or all), run command on each. + # TODO: Migrate callers to Check_Bash_Command_* ${index_list} = List_Indices_Or_All given_list=${member_index_list} : FOR ${index} IN @{index_list} \ Run_Bash_Command_On_Member command=${command} member_index=${index} Run_Bash_Command_On_Member [Arguments] ${command} ${member_index} - [Documentation] Obtain IP, call Utils and return output. This does not preserve active ssh session. - # TODO: Rename these keyword to Run_Bash_Command_On_Member to distinguish from Karaf (or even Windows) commands. + [Documentation] Obtain IP, call Utils and return output. This keeps previous ssh session active. + # TODO: Migrate callers to Check_Bash_Command_* ${member_ip} = Collections.Get_From_Dictionary dictionary=${ClusterManagement__index_to_ip_mapping} key=${member_index} ${output} = SSHKeywords.Run_Keyword_Preserve_Connection Utils.Run_Command_On_Controller ${member_ip} ${command} [Return] ${output} @@ -544,7 +566,6 @@ Install_Feature_On_Member With_Ssh_To_List_Or_All_Run_Keyword [Arguments] ${member_index_list} ${keyword_name} @{args} &{kwargs} [Documentation] For each index in given list (or all): activate SSH connection, run given Keyword, close active connection. Return None. - ... Note that if the Keyword affects SSH connections, results are still deterministic, but perhaps undesirable. ... Beware that in order to avoid "got positional argument after named arguments", first two arguments in the call should not be named. BuiltIn.Comment This keyword is experimental and there is high risk of being replaced by another approach. # TODO: For_Index_From_List_Or_All_Run_Keyword applied to With_Ssh_To_Member_Run_Keyword? @@ -552,9 +573,7 @@ With_Ssh_To_List_Or_All_Run_Keyword ${index_list} = List_Indices_Or_All given_list=${member_index_list} : FOR ${member_index} IN @{index_list} \ ${member_ip} = Resolve_IP_Address_For_Member ${member_index} - \ SSHKeywords.Open_Connection_To_Odl_System ip_address=${member_ip} - \ BuiltIn.Run_Keyword ${keyword_name} @{args} &{kwargs} - \ SSHLibrary.Close_Connection + \ SSHKeywords.Run_Unsafely_Keyword_Over_Temporary_Odl_Session ${member_ip} ${keyword_name} @{args} &{kwargs} Safe_With_Ssh_To_List_Or_All_Run_Keyword [Arguments] ${member_index_list} ${keyword_name} @{args} &{kwargs} @@ -562,11 +581,14 @@ Safe_With_Ssh_To_List_Or_All_Run_Keyword SSHKeywords.Run_Keyword_Preserve_Connection With_Ssh_To_List_Or_All_Run_Keyword ${member_index_list} ${keyword_name} @{args} &{kwargs} Clean_Directories_On_List_Or_All - [Arguments] ${member_index_list}=${EMPTY} ${directory_list}=${EMPTY} ${karaf_home}=${KARAF_HOME} + [Arguments] ${member_index_list}=${EMPTY} ${directory_list}=${EMPTY} ${karaf_home}=${KARAF_HOME} ${tmp_dir}=${EMPTY} [Documentation] Clear @{directory_list} or @{ODL_DEFAULT_DATA_PATHS} for members in given list or all. Return None. + ... If \${tmp_dir} is nonempty, use that location to preserve data/log/. ... This is intended to return Karaf (offline) to the state it was upon the first boot. ${path_list} = Builtin.Set Variable If "${directory_list}" == "${EMPTY}" ${ODL_DEFAULT_DATA_PATHS} ${directory_list} + BuiltIn.Run_Keyword_If """${tmp_dir}""" != "" Check_Bash_Command_On_List_Or_All mkdir -p '${tmp_dir}' && rm -vrf '${tmp_dir}/log' && mv -vf '${karaf_home}/data/log' '${tmp_dir}/' ${member_index_list} Safe_With_Ssh_To_List_Or_All_Run_Keyword ${member_index_list} ClusterManagement__Clean_Directories ${path_list} ${karaf_home} + BuiltIn.Run_Keyword_If """${tmp_dir}""" != "" Check_Bash_Command_On_List_Or_All mkdir -p '${karaf_home}/data' && rm -vrf '${karaf_home}/log' && mv -vf '${tmp_dir}/log' '${karaf_home}/data/' ${member_index_list} Store_Karaf_Log_On_List_Or_All [Arguments] ${member_index_list}=${EMPTY} ${dst_dir}=/tmp ${karaf_home}=${KARAF_HOME} diff --git a/csit/libraries/SSHKeywords.robot b/csit/libraries/SSHKeywords.robot index 91718c2f50..61f972c441 100644 --- a/csit/libraries/SSHKeywords.robot +++ b/csit/libraries/SSHKeywords.robot @@ -74,6 +74,22 @@ Run_Keyword_Preserve_Connection # Resource name has to be prepended, as KarafKeywords still contains a redirect. [Teardown] SSHKeywords.Restore_Current_SSH_Connection_From_Index ${current_connection.index} +Run_Keyword_With_Ssh + [Arguments] ${ip_address} ${keyword_name} @{args} &{kwargs} + [Documentation] Open temporary connection to given IP address, run keyword, close connection, restore previously active connection, return result. + Run_Keyword_Preserve_Connection Run_Unsafely_Keyword_Over_Temporary_Odl_Session ${ip_address} ${keyword_name} @{args} &{kwargs} + +Run_Unsafely_Keyword_Over_Temporary_Odl_Session + [Arguments] ${ip_address} ${keyword_name} @{args} &{kwargs} + [Documentation] Open connection to given IP address, run keyword, close connection, return result. + ... This is unsafe in the sense that previously active session will be switched out off, but safe in the sense only the temporary connection is closed. + Open_Connection_To_ODL_System ${ip_address} + # Not using Teardown, to avoid a call to close if the previous line fails. + ${status} ${result} = BuiltIn.Run_Keyword_And_Ignore_Error ${keyword_name} @{args} &{kwargs} + SSHLibrary.Close_Connection + BuiltIn.Return_From_Keyword_If "${status}" == "PASS" ${result} + BuiltIn.Fail ${result} + Log_Command_Results [Arguments] ${stdout} ${stderr} ${rc} [Documentation] Log everything returned by SSHLibrary.Execute_Command diff --git a/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot b/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot index a21a5be483..2caf4a17f4 100644 --- a/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot +++ b/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot @@ -26,9 +26,10 @@ Kill_All_Members ClusterManagement.Kill_Members_From_List_Or_All Unset_Tell_Based_Protocol_Usage - [Documentation] Comment out the flag usage in config file. - ClusterManagement.Run_Bash_Command_On_List_Or_All sed -ie "s/use-tell-based-protocol=/#use-tell-based-protocol=/g" ${DATASTORE_CFG} - ClusterManagement.Run_Bash_Command_On_List_Or_All cat ${DATASTORE_CFG} + [Documentation] Comment out the flag usage in config file. Also clean most data except data/log/. + ClusterManagement.Check_Bash_Command_On_List_Or_All sed -ie "s/use-tell-based-protocol=/#use-tell-based-protocol=/g" ${DATASTORE_CFG} + ClusterManagement.Check_Bash_Command_On_List_Or_All cat ${DATASTORE_CFG} + ClusterManagement.Clean_Directories_On_List_Or_All tmp_dir=/tmp Start_All_And_Sync [Documentation] Start each member and wait for sync. diff --git a/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_true.robot b/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_true.robot index 0474636001..3d40e18e06 100644 --- a/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_true.robot +++ b/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_true.robot @@ -7,8 +7,8 @@ Documentation Set tell-based protocol usage ... terms of the Eclipse Public License v1.0 which accompanies this distribution, ... and is available at http://www.eclipse.org/legal/epl-v10.html ... -... Suite stops all odl nodes, uncomment usage of tell-based protocol in -... config file and starts all nodes again. +... Suite stops all odl nodes, un-comment usage of tell-based protocol in +... config file (means make it true) and starts all nodes again. Suite Setup SetupUtils.Setup_Utils_For_Setup_And_Teardown Suite Teardown SSHLibrary.Close_All_Connections Library SSHLibrary @@ -26,9 +26,10 @@ Kill_All_Members ClusterManagement.Kill_Members_From_List_Or_All Set_Tell_Based_Protocol_Usage - [Documentation] Uncomment the flag usage line in config file - ClusterManagement.Run_Bash_Command_On_List_Or_All sed -ie "s/#use-tell-based-protocol=/use-tell-based-protocol=/g" ${DATASTORE_CFG} - ClusterManagement.Run_Bash_Command_On_List_Or_All cat ${DATASTORE_CFG} + [Documentation] Un-comment the flag usage in config file. Also clean most data except data/log/. + ClusterManagement.Check_Bash_Command_On_List_Or_All sed -ie "s/#use-tell-based-protocol=/use-tell-based-protocol=/g" ${DATASTORE_CFG} + ClusterManagement.Check_Bash_Command_On_List_Or_All cat ${DATASTORE_CFG} + ClusterManagement.Clean_Directories_On_List_Or_All tmp_dir=/tmp Start_All_And_Sync [Documentation] Start each member and wait for sync. diff --git a/csit/suites/test/cluster_reset.robot b/csit/suites/test/cluster_reset.robot index 1e71dae286..d9b17c9936 100644 --- a/csit/suites/test/cluster_reset.robot +++ b/csit/suites/test/cluster_reset.robot @@ -11,6 +11,8 @@ Documentation Kill nodes, delete all data created since boot, start nodes, w ... This suite is useful for undoing feature installation, Leader movement ... and for recovering from bably broken state. ... The intent is to provide speed compared to isolated job runs. +... +... FIXME: Import improvements from suites manipulating tell-based protocol setting. Suite Setup ClusterManagement.ClusterManagement_Setup Default Tags clustering critical Library DateTime