Implements partition and heal scenario.
Implements chasing the leader scenario.
Testplans included.
Signed-off-by: Peter Gubka <pgubka@cisco.com>
Change-Id: I6b6975b24ba1486ee46d30e690945bae2688383a
[Documentation] Rejoin isolated node.
ClusterManagement.Rejoin_Member_From_List_Or_All ${cs_isolated_index}
BuiltIn.Wait_Until_Keyword_Succeeds 5x 2s Verify_Singleton_Constant_On_Node ${cs_isolated_index} ${CS_CONSTANT_PREFIX}${cs_owner}
+
+Register_Flapping_Singleton_On_Nodes
+ [Arguments] ${index_list}
+ [Documentation] Register a candidate application on each node which starts the test.
+ : FOR ${index} IN @{index_list}
+ \ MdsalLowlevel.Register_Flapping_Singleton ${index}
+
+Unregister_Flapping_Singleton_On_Nodes_And_Validate_Results
+ [Arguments] ${index_list} ${rate_limit_to_pass} ${test_duration}
+ [Documentation] Unregister the testing service and check recevied statistics.
+ ${movements_count} = BuiltIn.Set_Variable ${0}
+ : FOR ${index} IN @{index_list}
+ \ ${count} = MdsalLowlevel.Unregister_Flapping_Singleton ${index}
+ \ BuiltIn.Run_Keyword_If ${count} < 0 BuiltIn.Fail No failure should have occured during the ${test_duration} timeout.
+ \ ${movements_count} = BuiltIn.Evaluate ${movements_count}+${count}
+ ${seconds} = DateTime.Convert_Time ${test_duration}
+ ${rate} = BuiltIn.Evaluate ${movements_count}/${seconds}
+ BuiltIn.Run_Keyword_If ${rate} < ${rate_limit_to_pass} BuiltIn.Fail Acceptance rate ${rate_limit_to_pass} not reached, actual rate is ${rate}.
...
... This test aims to establish the service operates correctly when faced with
... rapid application transitions without having a stabilized application.
-Suite Setup SetupUtils.Setup_Utils_For_Setup_And_Teardown
+Suite Setup Setup_Keyword
Suite Teardown SSHLibrary.Close_All_Connections
Test Setup SetupUtils.Setup_Test_With_Logging_And_Without_Fast_Failing
Test Teardown SetupUtils.Teardown_Test_Show_Bugs_If_Test_Failed
Default Tags critical
-Library Collections
-Library DateTime
Library SSHLibrary
-Resource ${CURDIR}/../../../libraries/MdsalLowlevel.robot
+Resource ${CURDIR}/../../../libraries/controller/CsCommon.robot
+Resource ${CURDIR}/../../../libraries/CompareStream.robot
Resource ${CURDIR}/../../../libraries/SetupUtils.robot
-Resource ${CURDIR}/../../../libraries/TemplatedRequests.robot
*** Variables ***
${TEST_DURATION} 1m
${ACCEPTED_PER_SEC_RATE} 100
+${ACCEPTED_PER_SEC_RATE_CARBON} 5
*** Test Cases ***
Register_Candidates
[Documentation] Register a candidate application on each node which starts the test.
- ${index_list} = ClusterManagement.List_All_Indices
- : FOR ${index} IN @{index_list}
- \ MdsalLowlevel.Register_Flapping_Singleton ${index}
+ CsCommon.Register_Flapping_Singleton_On_Nodes ${cs_all_indices}
Do_Nothing
[Documentation] Do nothing for the time of the test duration, because there is no api to monitor the statistics
Unregister_Candidates_And_Validate_Criteria
[Documentation] Unregister the testing service and check recevied statistics.
- ${index_list} = ClusterManagement.List_All_Indices
- ${movements_count} = BuiltIn.Set_Variable ${0}
- : FOR ${index} IN @{index_list}
- \ ${count} = MdsalLowlevel.Unregister_Flapping_Singleton ${index}
- \ BuiltIn.Run_Keyword_If ${count} < 0 BuiltIn.Fail No failure should have occured during the ${TEST_DURATION} timeout.
- \ ${movements_count} = BuiltIn.Evaluate ${movements_count}+${count}
- ${seconds} = DateTime.Convert_Time ${TEST_DURATION}
- ${rate} = BuiltIn.Evaluate ${movements_count}/${seconds}
- BuiltIn.Run_Keyword_If ${rate} < ${ACCEPTED_PER_SEC_RATE} BuiltIn.Fail Acceptance rate ${ACCEPTED_PER_SEC_RATE} not reached, actual rate is ${rate}.
+ ${rate_limit_to_pass} = CompareStream.Set_Variable_If_At_Most_Carbon ${ACCEPTED_PER_SEC_RATE_CARBON} ${ACCEPTED_PER_SEC_RATE}
+ CsCommon.Unregister_Flapping_Singleton_On_Nodes_And_Validate_Results ${cs_all_indices} ${rate_limit_to_pass} ${TEST_DURATION}
+
+*** Keywords ***
+Setup_Keyword
+ SetupUtils.Setup_Utils_For_Setup_And_Teardown
+ CsCommon.Cluster_Singleton_Init
--- /dev/null
+*** Settings ***
+Documentation Cluster Singleton testing: Chasing the Leader
+...
+... Copyright (c) 2017 Cisco Systems, Inc. and others. All rights reserved.
+...
+... This program and the accompanying materials are made available under the
+... terms of the Eclipse Public License v1.0 which accompanies this distribution,
+... and is available at http://www.eclipse.org/legal/epl-v10.html
+...
+... This test aims to establish the service operates correctly when faced with
+... rapid application transitions without having a stabilized application.
+Suite Setup Setup_Keyword
+Suite Teardown SSHLibrary.Close_All_Connections
+Test Setup SetupUtils.Setup_Test_With_Logging_And_Without_Fast_Failing
+Test Teardown SetupUtils.Teardown_Test_Show_Bugs_If_Test_Failed
+Default Tags critical
+Library SSHLibrary
+Resource ${CURDIR}/../../../libraries/controller/CsCommon.robot
+Resource ${CURDIR}/../../../libraries/CompareStream.robot
+Resource ${CURDIR}/../../../libraries/SetupUtils.robot
+
+*** Variables ***
+${TEST_DURATION} 24h
+${ACCEPTED_PER_SEC_RATE} 100
+${ACCEPTED_PER_SEC_RATE_CARBON} 5
+
+*** Test Cases ***
+Register_Candidates
+ [Documentation] Register a candidate application on each node which starts the test.
+ CsCommon.Register_Flapping_Singleton_On_Nodes ${cs_all_indices}
+
+Do_Nothing
+ [Documentation] Do nothing for the time of the test duration, because there is no api to monitor the statistics
+ ... during the test execution. Statistics are available only at the end, when unregister-flapping-singleton rpc is
+ ... called.
+ BuiltIn.Sleep ${TEST_DURATION}
+
+Unregister_Candidates_And_Validate_Criteria
+ [Documentation] Unregister the testing service and check recevied statistics.
+ ${rate_limit_to_pass} = CompareStream.Set_Variable_If_At_Most_Carbon ${ACCEPTED_PER_SEC_RATE_CARBON} ${ACCEPTED_PER_SEC_RATE}
+ CsCommon.Unregister_Flapping_Singleton_On_Nodes_And_Validate_Results ${cs_all_indices} ${rate_limit_to_pass} ${TEST_DURATION}
+
+*** Keywords ***
+Setup_Keyword
+ SetupUtils.Setup_Utils_For_Setup_And_Teardown
+ CsCommon.Cluster_Singleton_Init
--- /dev/null
+*** Settings ***
+Documentation Cluster Singleton testing: Partition And Heal longevity suite
+...
+... Copyright (c) 2017 Cisco Systems, Inc. and others. All rights reserved.
+...
+... This program and the accompanying materials are made available under the
+... terms of the Eclipse Public License v1.0 which accompanies this distribution,
+... and is available at http://www.eclipse.org/legal/epl-v10.html
+...
+... Cluster Singleton service is designed to ensure that only one instance of
+... an application is registered globally in the cluster.
+... The goal is to establish the service operates correctly in face of node
+... failures.
+Suite Setup Setup_Keyword
+Suite Teardown SSHLibrary.Close_All_Connections
+Test Setup SetupUtils.Setup_Test_With_Logging_And_Without_Fast_Failing
+Test Teardown SetupUtils.Teardown_Test_Show_Bugs_If_Test_Failed
+Default Tags critical
+Library Collections
+Library SSHLibrary
+Library RequestsLibrary
+Resource ${CURDIR}/../../../libraries/controller/CsCommon.robot
+Resource ${CURDIR}/../../../libraries/ClusterManagement.robot
+Resource ${CURDIR}/../../../libraries/MdsalLowlevel.robot
+Resource ${CURDIR}/../../../libraries/SetupUtils.robot
+Resource ${CURDIR}/../../../libraries/WaitForFailure.robot
+
+*** Variables ***
+${DURATION_24_HOURS_IN_SECONDS} 86400
+${STABILITY_TIMEOUT_ISOLATED} 120s
+${STABILITY_TIMEOUT_REJOINED} 60s
+@{STATUS_ISOLATED} ${501}
+
+*** Test Cases ***
+CS_Pertition_And_Heal
+ [Documentation] 24h lasting suite for isolating the cluster singleton leader repeatedly.
+ CsCommon.Register_Singleton_Constant_On_Nodes ${cs_all_indices}
+ WaitForFailure.Verify_Keyword_Does_Not_Fail_Within_Timeout ${DURATION_24_HOURS_IN_SECONDS} 3s Test_Scenario
+ CsCommon.Unregister_Singleton_Constant_On_Nodes ${cs_all_indices}
+
+*** Keywords ***
+Setup_Keyword
+ [Documentation] Suite setup.
+ SetupUtils.Setup_Utils_For_Setup_And_Teardown
+ CsCommon.Cluster_Singleton_Init
+
+Test_Scenario
+ [Documentation] Isolate the cluster node which is the owner, wait until the new owner is elected, then rejoin isolated node.
+ ... Monitor the stability of the singleton application and fail the the owner is changed during the monitoring. Monitoring
+ ... is done after the node isolation and after the node rejoin.
+ ${owner} ${candidates}= CsCommon.Get_And_Save_Present_CsOwner_And_CsCandidates 1
+ BuiltIn.Wait_Until_Keyword_Succeeds 6s 2s CsCommon.Verify_Singleton_Constant_On_Nodes ${cs_all_indices} ${CS_CONSTANT_PREFIX}${owner}
+ CsCommon.Isolate_Owner_And_Verify_Isolated
+ WaitForFailure.Verify_Keyword_Does_Not_Fail_Within_Timeout ${STABILITY_TIMEOUT_ISOLATED} 3s CsCommon.Verify_Singleton_Constant_During_Isolation
+ CsCommon.Rejoin_Node_And_Verify_Rejoined
+ WaitForFailure.Verify_Keyword_Does_Not_Fail_Within_Timeout ${STABILITY_TIMEOUT_REJOINED} 3s CsCommon.Verify_Singleton_Constant_On_Nodes ${cs_all_indices} ${CS_CONSTANT_PREFIX}${cs_owner}
--- /dev/null
+# Copyright (c) 2017 Cisco Systems, Inc. and others. All rights reserved.
+#
+# This program and the accompanying materials are made available under the
+# terms of the Eclipse Public License v1.0 which accompanies this distribution,
+# and is available at http://www.eclipse.org/legal/epl-v10.html
+
+# Place the suites in run order:
+integration/test/csit/suites/controller/cluster_singleton/chasing_the_leader_longevity.robot
--- /dev/null
+# Copyright (c) 2017 Cisco Systems, Inc. and others. All rights reserved.
+#
+# This program and the accompanying materials are made available under the
+# terms of the Eclipse Public License v1.0 which accompanies this distribution,
+# and is available at http://www.eclipse.org/legal/epl-v10.html
+
+# Place the suites in run order:
+integration/test/csit/suites/controller/cluster_singleton/partition_and_heal_longevity.robot
Code Review / integration / test.git / commitdiff