csit/suites/controller/cluster_singleton/partition_and_heal_longevity.robot

   1 *** Settings ***
   2 Documentation     Cluster Singleton testing: Partition And Heal longevity suite
   3 ...
   4 ...               Copyright (c) 2017 Cisco Systems, Inc. and others. All rights reserved.
   5 ...
   6 ...               This program and the accompanying materials are made available under the
   7 ...               terms of the Eclipse Public License v1.0 which accompanies this distribution,
   8 ...               and is available at http://www.eclipse.org/legal/epl-v10.html
   9 ...
  10 ...               Cluster Singleton service is designed to ensure that only one instance of
  11 ...               an application is registered globally in the cluster.
  12 ...               The goal is to establish the service operates correctly in face of node
  13 ...               failures.
  14 Suite Setup       Setup_Keyword
  15 Suite Teardown    SSHLibrary.Close_All_Connections
  16 Test Setup        SetupUtils.Setup_Test_With_Logging_And_Without_Fast_Failing
  17 Test Teardown     SetupUtils.Teardown_Test_Show_Bugs_If_Test_Failed
  18 Default Tags      critical
  19 Library           Collections
  20 Library           SSHLibrary
  21 Library           RequestsLibrary
  22 Resource          ${CURDIR}/../../../libraries/controller/CsCommon.robot
  23 Resource          ${CURDIR}/../../../libraries/ClusterManagement.robot
  24 Resource          ${CURDIR}/../../../libraries/MdsalLowlevel.robot
  25 Resource          ${CURDIR}/../../../libraries/SetupUtils.robot
  26 Resource          ${CURDIR}/../../../libraries/WaitForFailure.robot
  27
  28 *** Variables ***
  29 ${DURATION_24_HOURS_IN_SECONDS}    86400
  30 ${STABILITY_TIMEOUT_ISOLATED}    120s
  31 ${STABILITY_TIMEOUT_REJOINED}    60s
  32 @{STATUS_ISOLATED}    ${501}
  33
  34 *** Test Cases ***
  35 CS_Pertition_And_Heal
  36     [Documentation]    24h lasting suite for isolating the cluster singleton leader repeatedly.
  37     CsCommon.Register_Singleton_Constant_On_Nodes    ${cs_all_indices}
  38     WaitForFailure.Verify_Keyword_Does_Not_Fail_Within_Timeout    ${DURATION_24_HOURS_IN_SECONDS}    3s    Test_Scenario
  39     CsCommon.Unregister_Singleton_Constant_On_Nodes    ${cs_all_indices}
  40
  41 *** Keywords ***
  42 Setup_Keyword
  43     [Documentation]    Suite setup.
  44     SetupUtils.Setup_Utils_For_Setup_And_Teardown    http_timeout=30
  45     CsCommon.Cluster_Singleton_Init
  46
  47 Test_Scenario
  48     [Documentation]    Isolate the cluster node which is the owner, wait until the new owner is elected, then rejoin isolated node.
  49     ...    Monitor the stability of the singleton application and fail the the owner is changed during the monitoring. Monitoring
  50     ...    is done after the node isolation and after the node rejoin.
  51     ${owner}    ${candidates}=    CsCommon.Get_And_Save_Present_CsOwner_And_CsCandidates    1
  52     BuiltIn.Wait_Until_Keyword_Succeeds    6s    2s    CsCommon.Verify_Singleton_Constant_On_Nodes    ${cs_all_indices}    ${CS_CONSTANT_PREFIX}${owner}
  53     CsCommon.Isolate_Owner_And_Verify_Isolated
  54     WaitForFailure.Verify_Keyword_Does_Not_Fail_Within_Timeout    ${STABILITY_TIMEOUT_ISOLATED}    3s    CsCommon.Verify_Singleton_Constant_During_Isolation
  55     CsCommon.Rejoin_Node_And_Verify_Rejoined
  56     WaitForFailure.Verify_Keyword_Does_Not_Fail_Within_Timeout    ${STABILITY_TIMEOUT_REJOINED}    3s    CsCommon.Verify_Singleton_Constant_On_Nodes    ${cs_all_indices}    ${CS_CONSTANT_PREFIX}${cs_owner}