Clean data when switching tell-base protocol. 74/56474/8
authorVratko Polak <vrpolak@cisco.com>
Thu, 4 May 2017 13:03:38 +0000 (15:03 +0200)
committerVratko Polak <vrpolak@cisco.com>
Thu, 4 May 2017 13:03:38 +0000 (15:03 +0200)
That is to make sure persisted data does not create failures.
That can be reverted when suite results are stable enough.

SSHKeywords.Run_Keyword_With_Ssh added to be more defensive
against situation when inner failure prevents session restore.

ClusterManagement.Check_Bash_Command_On_List_Or_All added
to rely on this defensivenes when running bash commands
with logging and checks.

Change-Id: I9ac004324f06646aec582987cd4665a61db88b4f
Signed-off-by: Vratko Polak <vrpolak@cisco.com>
csit/libraries/ClusterManagement.robot
csit/libraries/SSHKeywords.robot
csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot
csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_true.robot
csit/suites/test/cluster_reset.robot

index 473d8590576dba58290992404e2c697721411920..23c79de01e95ce162f6a009892f6c3a827e287a0 100644 (file)
@@ -489,17 +489,39 @@ Flush_Iptables_From_List_Or_All
     ${command} =    BuiltIn.Set_Variable    sudo iptables -v -F
     ${output} =    Run_Bash_Command_On_List_Or_All    command=${command}    member_index_list=${member_index_list}
 
+Check_Bash_Command_On_List_Or_All
+    [Arguments]    ${command}    ${member_index_list}=${EMPTY}    ${return_success_only}=False    ${log_on_success}=True    ${log_on_failure}=True    ${stderr_must_be_empty}=True
+    [Documentation]    Cycle through indices (or all), run bash command on each, using temporary SSH session and restoring the previously active one.
+    ${index_list} =    List_Indices_Or_All    given_list=${member_index_list}
+    : FOR    ${index}    IN    @{index_list}
+    \    Check_Bash_Command_On_Member    command=${command}    member_index=${index}    return_success_only=${return_success_only}    log_on_success=${log_on_success}    log_on_failure=${log_on_failure}
+    \    ...    stderr_must_be_empty=${stderr_must_be_empty}
+
+Check_Bash_Command_On_Member
+    [Arguments]    ${command}    ${member_index}    ${return_success_only}=False    ${log_on_success}=True    ${log_on_failure}=True    ${stderr_must_be_empty}=True
+    [Documentation]    Open SSH session, call SSHKeywords.Execute_Command_Passes, close session, restore previously active session and return output.
+    BuiltIn.Run_Keyword_And_Return    SSHKeywords.Run_Keyword_Preserve_Connection    Check_Unsafely_Bash_Command_On_Member    ${command}    ${member_index}    return_success_only=${return_success_only}    log_on_success=${log_on_success}
+    ...    log_on_failure=${log_on_failure}    stderr_must_be_empty=${stderr_must_be_empty}
+
+Check_Unsafely_Bash_Command_On_Member
+    [Arguments]    ${command}    ${member_index}    ${return_success_only}=False    ${log_on_success}=True    ${log_on_failure}=True    ${stderr_must_be_empty}=True
+    [Documentation]    Obtain Ip address, open session, call SSHKeywords.Execute_Command_Passes, close session and return output. This affects which SSH session is active.
+    ${member_ip} =    Resolve_Ip_Address_For_Member    ${member_index}
+    BuiltIn.Run_Keyword_And_Return    SSHKeywords.Run_Unsafely_Keyword_Over_Temporary_Odl_Session    ${member_ip}    Execute_Command_Passes    ${command}    return_success_only=${return_success_only}    log_on_success=${log_on_success}
+    ...    log_on_failure=${log_on_failure}    stderr_must_be_empty=${stderr_must_be_empty}
+
 Run_Bash_Command_On_List_Or_All
     [Arguments]    ${command}    ${member_index_list}=${EMPTY}
     [Documentation]    Cycle through indices (or all), run command on each.
+    # TODO: Migrate callers to Check_Bash_Command_*
     ${index_list} =    List_Indices_Or_All    given_list=${member_index_list}
     : FOR    ${index}    IN    @{index_list}
     \    Run_Bash_Command_On_Member    command=${command}    member_index=${index}
 
 Run_Bash_Command_On_Member
     [Arguments]    ${command}    ${member_index}
-    [Documentation]    Obtain IP, call Utils and return output. This does not preserve active ssh session.
-    # TODO: Rename these keyword to Run_Bash_Command_On_Member to distinguish from Karaf (or even Windows) commands.
+    [Documentation]    Obtain IP, call Utils and return output. This keeps previous ssh session active.
+    # TODO: Migrate callers to Check_Bash_Command_*
     ${member_ip} =    Collections.Get_From_Dictionary    dictionary=${ClusterManagement__index_to_ip_mapping}    key=${member_index}
     ${output} =    SSHKeywords.Run_Keyword_Preserve_Connection    Utils.Run_Command_On_Controller    ${member_ip}    ${command}
     [Return]    ${output}
@@ -544,7 +566,6 @@ Install_Feature_On_Member
 With_Ssh_To_List_Or_All_Run_Keyword
     [Arguments]    ${member_index_list}    ${keyword_name}    @{args}    &{kwargs}
     [Documentation]    For each index in given list (or all): activate SSH connection, run given Keyword, close active connection. Return None.
-    ...    Note that if the Keyword affects SSH connections, results are still deterministic, but perhaps undesirable.
     ...    Beware that in order to avoid "got positional argument after named arguments", first two arguments in the call should not be named.
     BuiltIn.Comment    This keyword is experimental and there is high risk of being replaced by another approach.
     # TODO: For_Index_From_List_Or_All_Run_Keyword applied to With_Ssh_To_Member_Run_Keyword?
@@ -552,9 +573,7 @@ With_Ssh_To_List_Or_All_Run_Keyword
     ${index_list} =    List_Indices_Or_All    given_list=${member_index_list}
     : FOR    ${member_index}    IN    @{index_list}
     \    ${member_ip} =    Resolve_IP_Address_For_Member    ${member_index}
-    \    SSHKeywords.Open_Connection_To_Odl_System    ip_address=${member_ip}
-    \    BuiltIn.Run_Keyword    ${keyword_name}    @{args}    &{kwargs}
-    \    SSHLibrary.Close_Connection
+    \    SSHKeywords.Run_Unsafely_Keyword_Over_Temporary_Odl_Session    ${member_ip}    ${keyword_name}    @{args}    &{kwargs}
 
 Safe_With_Ssh_To_List_Or_All_Run_Keyword
     [Arguments]    ${member_index_list}    ${keyword_name}    @{args}    &{kwargs}
@@ -562,11 +581,14 @@ Safe_With_Ssh_To_List_Or_All_Run_Keyword
     SSHKeywords.Run_Keyword_Preserve_Connection    With_Ssh_To_List_Or_All_Run_Keyword    ${member_index_list}    ${keyword_name}    @{args}    &{kwargs}
 
 Clean_Directories_On_List_Or_All
-    [Arguments]    ${member_index_list}=${EMPTY}    ${directory_list}=${EMPTY}    ${karaf_home}=${KARAF_HOME}
+    [Arguments]    ${member_index_list}=${EMPTY}    ${directory_list}=${EMPTY}    ${karaf_home}=${KARAF_HOME}    ${tmp_dir}=${EMPTY}
     [Documentation]    Clear @{directory_list} or @{ODL_DEFAULT_DATA_PATHS} for members in given list or all. Return None.
+    ...    If \${tmp_dir} is nonempty, use that location to preserve data/log/.
     ...    This is intended to return Karaf (offline) to the state it was upon the first boot.
     ${path_list} =    Builtin.Set Variable If    "${directory_list}" == "${EMPTY}"    ${ODL_DEFAULT_DATA_PATHS}    ${directory_list}
+    BuiltIn.Run_Keyword_If    """${tmp_dir}""" != ""    Check_Bash_Command_On_List_Or_All    mkdir -p '${tmp_dir}' && rm -vrf '${tmp_dir}/log' && mv -vf '${karaf_home}/data/log' '${tmp_dir}/'    ${member_index_list}
     Safe_With_Ssh_To_List_Or_All_Run_Keyword    ${member_index_list}    ClusterManagement__Clean_Directories    ${path_list}    ${karaf_home}
+    BuiltIn.Run_Keyword_If    """${tmp_dir}""" != ""    Check_Bash_Command_On_List_Or_All    mkdir -p '${karaf_home}/data' && rm -vrf '${karaf_home}/log' && mv -vf '${tmp_dir}/log' '${karaf_home}/data/'    ${member_index_list}
 
 Store_Karaf_Log_On_List_Or_All
     [Arguments]    ${member_index_list}=${EMPTY}    ${dst_dir}=/tmp    ${karaf_home}=${KARAF_HOME}
index 91718c2f50f844cee03b4b4e052b9b0cc42128e1..61f972c44110b2fb6f93a74c75c5eb2f48dd7856 100644 (file)
@@ -74,6 +74,22 @@ Run_Keyword_Preserve_Connection
     # Resource name has to be prepended, as KarafKeywords still contains a redirect.
     [Teardown]    SSHKeywords.Restore_Current_SSH_Connection_From_Index    ${current_connection.index}
 
+Run_Keyword_With_Ssh
+    [Arguments]    ${ip_address}    ${keyword_name}    @{args}    &{kwargs}
+    [Documentation]    Open temporary connection to given IP address, run keyword, close connection, restore previously active connection, return result.
+    Run_Keyword_Preserve_Connection    Run_Unsafely_Keyword_Over_Temporary_Odl_Session    ${ip_address}    ${keyword_name}    @{args}    &{kwargs}
+
+Run_Unsafely_Keyword_Over_Temporary_Odl_Session
+    [Arguments]    ${ip_address}    ${keyword_name}    @{args}    &{kwargs}
+    [Documentation]    Open connection to given IP address, run keyword, close connection, return result.
+    ...    This is unsafe in the sense that previously active session will be switched out off, but safe in the sense only the temporary connection is closed.
+    Open_Connection_To_ODL_System    ${ip_address}
+    # Not using Teardown, to avoid a call to close if the previous line fails.
+    ${status}    ${result} =    BuiltIn.Run_Keyword_And_Ignore_Error    ${keyword_name}    @{args}    &{kwargs}
+    SSHLibrary.Close_Connection
+    BuiltIn.Return_From_Keyword_If    "${status}" == "PASS"    ${result}
+    BuiltIn.Fail    ${result}
+
 Log_Command_Results
     [Arguments]    ${stdout}    ${stderr}    ${rc}
     [Documentation]    Log everything returned by SSHLibrary.Execute_Command
index a21a5be483d3c8cf0c63e5f22376850d2a059c5d..2caf4a17f4357c9f0e2bf1edc3b0386d844b9c1c 100644 (file)
@@ -26,9 +26,10 @@ Kill_All_Members
     ClusterManagement.Kill_Members_From_List_Or_All
 
 Unset_Tell_Based_Protocol_Usage
-    [Documentation]    Comment out the flag usage in config file.
-    ClusterManagement.Run_Bash_Command_On_List_Or_All    sed -ie "s/use-tell-based-protocol=/#use-tell-based-protocol=/g" ${DATASTORE_CFG}
-    ClusterManagement.Run_Bash_Command_On_List_Or_All    cat ${DATASTORE_CFG}
+    [Documentation]    Comment out the flag usage in config file. Also clean most data except data/log/.
+    ClusterManagement.Check_Bash_Command_On_List_Or_All    sed -ie "s/use-tell-based-protocol=/#use-tell-based-protocol=/g" ${DATASTORE_CFG}
+    ClusterManagement.Check_Bash_Command_On_List_Or_All    cat ${DATASTORE_CFG}
+    ClusterManagement.Clean_Directories_On_List_Or_All    tmp_dir=/tmp
 
 Start_All_And_Sync
     [Documentation]    Start each member and wait for sync.
index 0474636001a2197bb51abf1ffd6523b635dc2c5f..3d40e18e061436383fb23b4e3a36b476891f1869 100644 (file)
@@ -7,8 +7,8 @@ Documentation     Set tell-based protocol usage
 ...               terms of the Eclipse Public License v1.0 which accompanies this distribution,
 ...               and is available at http://www.eclipse.org/legal/epl-v10.html
 ...
-...               Suite stops all odl nodes, uncomment usage of tell-based protocol in
-...               config file and starts all nodes again.
+...               Suite stops all odl nodes, un-comment usage of tell-based protocol in
+...               config file (means make it true) and starts all nodes again.
 Suite Setup       SetupUtils.Setup_Utils_For_Setup_And_Teardown
 Suite Teardown    SSHLibrary.Close_All_Connections
 Library           SSHLibrary
@@ -26,9 +26,10 @@ Kill_All_Members
     ClusterManagement.Kill_Members_From_List_Or_All
 
 Set_Tell_Based_Protocol_Usage
-    [Documentation]    Uncomment the flag usage line in config file
-    ClusterManagement.Run_Bash_Command_On_List_Or_All    sed -ie "s/#use-tell-based-protocol=/use-tell-based-protocol=/g" ${DATASTORE_CFG}
-    ClusterManagement.Run_Bash_Command_On_List_Or_All    cat ${DATASTORE_CFG}
+    [Documentation]    Un-comment the flag usage in config file. Also clean most data except data/log/.
+    ClusterManagement.Check_Bash_Command_On_List_Or_All    sed -ie "s/#use-tell-based-protocol=/use-tell-based-protocol=/g" ${DATASTORE_CFG}
+    ClusterManagement.Check_Bash_Command_On_List_Or_All    cat ${DATASTORE_CFG}
+    ClusterManagement.Clean_Directories_On_List_Or_All    tmp_dir=/tmp
 
 Start_All_And_Sync
     [Documentation]    Start each member and wait for sync.
index 1e71dae286a3475055b7a62e806743c3a7d91ade..d9b17c99360d2b5865635445a883b993d8070592 100644 (file)
@@ -11,6 +11,8 @@ Documentation     Kill nodes, delete all data created since boot, start nodes, w
 ...               This suite is useful for undoing feature installation, Leader movement
 ...               and for recovering from bably broken state.
 ...               The intent is to provide speed compared to isolated job runs.
+...
+...               FIXME: Import improvements from suites manipulating tell-based protocol setting.
 Suite Setup       ClusterManagement.ClusterManagement_Setup
 Default Tags      clustering    critical
 Library           DateTime