csit/suites/controller/cluster_singleton/partition_and_heal_longevity.robot

   1 *** Settings ***
   2 Documentation       Cluster Singleton testing: Partition And Heal longevity suite
   3 ...
   4 ...                 Copyright (c) 2017 Cisco Systems, Inc. and others. All rights reserved.
   5 ...
   6 ...                 This program and the accompanying materials are made available under the
   7 ...                 terms of the Eclipse Public License v1.0 which accompanies this distribution,
   8 ...                 and is available at http://www.eclipse.org/legal/epl-v10.html
   9 ...
  10 ...                 Cluster Singleton service is designed to ensure that only one instance of
  11 ...                 an application is registered globally in the cluster.
  12 ...                 The goal is to establish the service operates correctly in face of node
  13 ...                 failures.
  14
  15 Library             Collections
  16 Library             SSHLibrary
  17 Library             RequestsLibrary
  18 Resource            ${CURDIR}/../../../libraries/controller/CsCommon.robot
  19 Resource            ${CURDIR}/../../../libraries/ClusterManagement.robot
  20 Resource            ${CURDIR}/../../../libraries/MdsalLowlevel.robot
  21 Resource            ${CURDIR}/../../../libraries/SetupUtils.robot
  22 Resource            ${CURDIR}/../../../libraries/WaitForFailure.robot
  23
  24 Suite Setup         Setup_Keyword
  25 Suite Teardown      SSHLibrary.Close_All_Connections
  26 Test Setup          SetupUtils.Setup_Test_With_Logging_And_Without_Fast_Failing
  27 Test Teardown       SetupUtils.Teardown_Test_Show_Bugs_If_Test_Failed
  28
  29 Default Tags        critical
  30
  31
  32 *** Variables ***
  33 # TODO: change back to 24h when releng has more granular steps to kill VMs than days; now 23h=82800s
  34 ${LONGEVITY_TEST_DURATION_IN_SECS}      82800
  35 ${STABILITY_TIMEOUT_ISOLATED}           120s
  36 ${STABILITY_TIMEOUT_REJOINED}           60s
  37 @{STATUS_ISOLATED}                      ${501}
  38
  39
  40 *** Test Cases ***
  41 CS_Pertition_And_Heal
  42     [Documentation]    24h lasting suite for isolating the cluster singleton leader repeatedly.
  43     CsCommon.Register_Singleton_Constant_On_Nodes    ${cs_all_indices}
  44     WaitForFailure.Verify_Keyword_Does_Not_Fail_Within_Timeout
  45     ...    ${LONGEVITY_TEST_DURATION_IN_SECS}
  46     ...    3s
  47     ...    Test_Scenario
  48     CsCommon.Unregister_Singleton_Constant_On_Nodes    ${cs_all_indices}
  49
  50
  51 *** Keywords ***
  52 Setup_Keyword
  53     [Documentation]    Suite setup.
  54     SetupUtils.Setup_Utils_For_Setup_And_Teardown    http_timeout=125
  55     CsCommon.Cluster_Singleton_Init
  56
  57 Test_Scenario
  58     [Documentation]    Isolate the cluster node which is the owner, wait until the new owner is elected, then rejoin isolated node.
  59     ...    Monitor the stability of the singleton application and fail the the owner is changed during the monitoring. Monitoring
  60     ...    is done after the node isolation and after the node rejoin.
  61     ${owner}    ${candidates}=    CsCommon.Get_And_Save_Present_CsOwner_And_CsCandidates    1
  62     BuiltIn.Wait_Until_Keyword_Succeeds
  63     ...    15s
  64     ...    2s
  65     ...    CsCommon.Verify_Singleton_Constant_On_Nodes
  66     ...    ${cs_all_indices}
  67     ...    ${CS_CONSTANT_PREFIX}${owner}
  68     KarafKeywords.Log_Message_To_Controller_Karaf    Isolating node${owner}.
  69     CsCommon.Isolate_Owner_And_Verify_Isolated
  70     WaitForFailure.Verify_Keyword_Does_Not_Fail_Within_Timeout
  71     ...    ${STABILITY_TIMEOUT_ISOLATED}
  72     ...    3s
  73     ...    CsCommon.Verify_Singleton_Constant_During_Isolation
  74     KarafKeywords.Log_Message_To_Controller_Karaf    Rejoining node${owner}.
  75     CsCommon.Rejoin_Node_And_Verify_Rejoined
  76     WaitForFailure.Verify_Keyword_Does_Not_Fail_Within_Timeout
  77     ...    ${STABILITY_TIMEOUT_REJOINED}
  78     ...    3s
  79     ...    CsCommon.Verify_Singleton_Constant_On_Nodes
  80     ...    ${cs_all_indices}
  81     ...    ${CS_CONSTANT_PREFIX}${cs_owner}