csit/suites/controller/dom_rpc_broker/rpc_provider_partition_and_heal_longevity.robot

   1 *** Settings ***
   2 Documentation       DOMRpcBroker testing: RPC Provider Partition And Heal
   3 ...
   4 ...                 Copyright (c) 2017 Cisco Systems, Inc. and others. All rights reserved.
   5 ...
   6 ...                 This program and the accompanying materials are made available under the
   7 ...                 terms of the Eclipse Public License v1.0 which accompanies this distribution,
   8 ...                 and is available at http://www.eclipse.org/legal/epl-v10.html
   9 ...
  10 ...                 This tests establishes that the RPC service operates correctly when faced
  11 ...                 with node failures.
  12 ...                 This suite supports more than three node cluster setup too.
  13
  14 Library             SSHLibrary
  15 Resource            ${CURDIR}/../../../libraries/controller/DrbCommons.robot
  16 Resource            ${CURDIR}/../../../libraries/SetupUtils.robot
  17 Resource            ${CURDIR}/../../../libraries/WaitForFailure.robot
  18
  19 Suite Setup         Setup_Kw
  20 Suite Teardown      SSHLibrary.Close_All_Connections
  21 Test Setup          SetupUtils.Setup_Test_With_Logging_And_Without_Fast_Failing
  22 Test Teardown       SetupUtils.Teardown_Test_Show_Bugs_If_Test_Failed
  23
  24 Default Tags        critical
  25
  26
  27 *** Variables ***
  28 @{INSTALLED_RPC_MEMEBER_IDX_LIST}       ${1}    ${2}
  29 # TODO: Consider unregistering one at random for each iteration, as in the precedence longevity suite.
  30 ${TESTED_MEMBER_WITHOUT_RPC_IDX}        ${3}
  31 # TODO: change back to 24h when releng has more granular steps to kill VMs than days; now 23h=82800s
  32 ${LONGEVITY_TEST_DURATION_IN_SECS}      82800
  33
  34
  35 *** Test Cases ***
  36 Rpc_Provider_Precedence_Longevity
  37     [Documentation]    Test register rpc on two of three nodes and repeat the tested scenario for 24h.
  38     DrbCommons.Register_Rpc_On_Nodes    ${INSTALLED_RPC_MEMEBER_IDX_LIST}
  39     WaitForFailure.Verify_Keyword_Does_Not_Fail_Within_Timeout
  40     ...    ${LONGEVITY_TEST_DURATION_IN_SECS}
  41     ...    1s
  42     ...    Test_Scenario
  43     DrbCommons.Unregister_Rpc_On_Nodes    ${INSTALLED_RPC_MEMEBER_IDX_LIST}
  44
  45
  46 *** Keywords ***
  47 Setup_Kw
  48     [Documentation]    Setup keyword. Create ${possible_constants} list with possible variables of remote constants.
  49     SetupUtils.Setup_Utils_For_Setup_And_Teardown    http_timeout=125
  50     DrbCommons.DrbCommons_Init
  51
  52 Test_Scenario
  53     [Documentation]    Isolate and rejoin one of the registrated nodes while testing expected constants.
  54     DrbCommons.Verify_Constant_On_Active_Nodes
  55     ${isolated_idx} =    DrbCommons.Get_Constant_Index_From_Node    ${TESTED_MEMBER_WITHOUT_RPC_IDX}
  56     DrbCommons.Isolate_Node    ${isolated_idx}
  57     BuiltIn.Wait_Until_Keyword_Succeeds    60s    5s    DrbCommons.Verify_Constant_On_Active_Nodes
  58     DrbCommons.Rejoin_Node    ${isolated_idx}
  59     WaitForFailure.Verify_Keyword_Does_Not_Fail_Within_Timeout
  60     ...    20s
  61     ...    1s
  62     ...    DrbCommons.Verify_Constant_On_Active_Nodes