From: Peter Gubka Date: Tue, 24 Jan 2017 10:23:49 +0000 (+0100) Subject: Add suite for global singleton rpc testing X-Git-Tag: release/boron-sr3~148 X-Git-Url: https://git.opendaylight.org/gerrit/gitweb?a=commitdiff_plain;h=20528cbc6d95c3b32a3248394b6c8247040a666b;p=integration%2Ftest.git Add suite for global singleton rpc testing Change-Id: I7f3d198e0806b12c21fb75261d6da53025417869 Signed-off-by: Peter Gubka --- diff --git a/csit/libraries/SetupUtils.robot b/csit/libraries/SetupUtils.robot index eccae957d0..c748097a1f 100644 --- a/csit/libraries/SetupUtils.robot +++ b/csit/libraries/SetupUtils.robot @@ -41,6 +41,12 @@ Setup_Test_With_Logging_And_Without_Fast_Failing FailFast.Run_Even_When_Failing_Fast BuiltIn.Run Keyword And Ignore Error KarafKeywords.Log_Testcase_Start_To_Controller_Karaf +Setup_Logging_For_Debug_Purposes_On_List_Or_All + [Arguments] ${log_level} ${loggers_list} ${member_index_list}=${EMPTY} + [Documentation] Set the log level for given loggers on node nodes of the cluster + : FOR ${logger} IN @{loggers_list} + \ ClusterManagement.Run_Karaf_Command_On_List_Or_All log:set ${log_level} ${logger} member_index_list=${member_index_list} + Set_Known_Bug_Id [Arguments] ${id} [Documentation] Tell the Teardown keywords that any failure from now on is due to the specified known bug. diff --git a/csit/suites/controller/singleton_service/global_rpc_isolate.robot b/csit/suites/controller/singleton_service/global_rpc_isolate.robot new file mode 100644 index 0000000000..a1960fb407 --- /dev/null +++ b/csit/suites/controller/singleton_service/global_rpc_isolate.robot @@ -0,0 +1,122 @@ +*** Settings *** +Documentation Controller functional HA testing of global singleton rpcs. +... +... Copyright (c) 2017 Cisco Systems, Inc. and others. All rights reserved. +... +... This program and the accompanying materials are made available under the +... terms of the Eclipse Public License v1.0 which accompanies this distribution, +... and is available at http://www.eclipse.org/legal/epl-v10.html +Suite Setup Setup_Suite +Suite Teardown SSHLibrary.Close_All_Connections +Test Setup SetupUtils.Setup_Test_With_Logging_And_Without_Fast_Failing +Test Teardown SetupUtils.Teardown_Test_Show_Bugs_If_Test_Failed +Default Tags critical +Resource ${CURDIR}/../../../libraries/ClusterManagement.robot +Resource ${CURDIR}/../../../libraries/KarafKeywords.robot +Resource ${CURDIR}/../../../libraries/SetupUtils.robot +Resource ${CURDIR}/../../../libraries/TemplatedRequests.robot +Resource ${CURDIR}/../../../variables/Variables.robot +Resource ${CURDIR}/../../../libraries/WaitForFailure.robot + +*** Variables *** +${RPC_URL} /restconf/operations/basic-rpc-test:basic-global +&{EMPTY_DICT} +${SERVICE} Basic-rpc-test'] +${TEST_LOG_LEVEL} info +@{TEST_LOG_COMPONENTS} org.opendaylight.controller.remote.rpc +${EOS_URL} /restconf/operational/entity-owners:entity-owners +${RPC_STATUS_ISOLATED} 501 +@{NO_TAGS} + +*** Test Cases *** +Get_Basic_Rpc_Test_Owner + [Documentation] Find a service owner and successors. + [Tags] @{NO_TAGS} + Get_Present_Brt_Owner_And_Successors 1 store=${True} + +Rpc_Before_Isolation_On_Owner + [Documentation] Run rpc on the service owner. + Run_Rpc ${brt_owner} + +Rpc_Before_Isolation_On_Successors + [Documentation] Run rpc on non owher cluster nodes. + : FOR ${idx} IN @{brt_successors} + \ Run_Rpc ${idx} + +Isolate_Current_Owner_Member + [Documentation] Isolating cluster node which is the owner. + [Tags] @{NO_TAGS} + ClusterManagement.Isolate_Member_From_List_Or_All ${brt_owner} + BuiltIn.Set Suite variable ${old_brt_owner} ${brt_owner} + BuiltIn.Set Suite variable ${old_brt_successors} ${brt_successors} + +Verify_New_Basic_Rpc_Test_Owner_Elected + [Documentation] Verify new owner of the service is elected. + ${idx}= Collections.Get_From_List ${old_brt_successors} 0 + BuiltIn.Wait_Until_Keyword_Succeeds 5x 2s Verify_Owner_Elected ${True} ${old_brt_owner} ${idx} + Get_Present_Brt_Owner_And_Successors ${idx} store=${True} + +Rpc_On_Isolated_Node + [Documentation] Run rpc on isolated cluster node. + ${session} = Resolve_Http_Session_For_Member member_index=${old_brt_owner} + ${out} = TemplatedRequests.Get_From_Uri ${EOS_URL} session=${session} + KarafKeywords.Log_Message_To_Controller_Karaf EOS rest resp: ${out} + ${resp} = RequestsLibrary.Post Request ${session} ${RPC_URL} data=${EMPTY} + BuiltIn.Should_Be_Equal_As_Numbers ${resp.status_code} ${RPC_STATUS_ISOLATED} + +Rpc_On_Non_Isolated_Cluster_Nodes + [Documentation] Run rpc on remained cluster nodes. + : FOR ${idx} IN @{old_brt_successors} + \ Run_Rpc ${idx} + +Rejoin_Isolated_Member + [Documentation] Rejoin isolated node + [Tags] @{NO_TAGS} + ClusterManagement.Rejoin_Member_From_List_Or_All ${old_brt_owner} + +Verify_New_Owner_Remained_After_Rejoin + [Documentation] Verify no owner change happened after rejoin. + WaitForFailure.Verify_Keyword_Does_Not_Fail_Within_Timeout 15s 2s Verify_Owner_Elected ${False} ${brt_owner} ${brt_owner} + +Rpc_After_Rejoin_On_New_Owner + [Documentation] Run rpc on the new service owner node. + Run_Rpc ${brt_owner} + +Rpc_After_Rejoin_On_Old_Owner + [Documentation] Run rpc on rejoined cluster node. + Run_Rpc ${old_brt_owner} + +Rpc_After_Rejoin_On_All + [Documentation] Run rpc again on all nodes. + Run_Rpc ${brt_owner} + : FOR ${idx} IN @{brt_successors} + \ Run_Rpc ${idx} + +*** Keywords *** +Setup_Suite + SetupUtils.Setup_Utils_For_Setup_And_Teardown + SetupUtils.Setup_Logging_For_Debug_Purposes_On_List_Or_All ${TEST_LOG_LEVEL} ${TEST_LOG_COMPONENTS} + +Run_Rpc + [Arguments] ${node_idx} + [Documentation] Run rpc and log the entity ownership service details to karaf log. + ... Logging the details was a developer's request during the implementation to improve debugging. + ${session} = Resolve_Http_Session_For_Member member_index=${node_idx} + ${out} = TemplatedRequests.Get_From_Uri ${EOS_URL} session=${session} + KarafKeywords.Log_Message_To_Controller_Karaf EOS rest resp: ${out} + TemplatedRequests.Post_To_Uri ${RPC_URL} ${EMPTY} ${EMPTY_DICT} ${EMPTY_DICT} session=${session} + +Verify_Owner_Elected + [Arguments] ${new_elected} ${old_owner} ${node_to_ask} + [Documentation] Verify new owner was elected or remained the same. + ${owner} ${successors}= Get_Present_Brt_Owner_And_Successors ${node_to_ask} + BuiltIn.Run_Keyword_If ${new_elected} BuiltIn.Should_Not_Be_Equal_As_Numbers ${old_owner} ${owner} + BuiltIn.Run_Keyword_Unless ${new_elected} BuiltIn.Should_Be_Equal_As_numbers ${old_owner} ${owner} + +Get_Present_Brt_Owner_And_Successors + [Arguments] ${node_to_ask} ${store}=${False} + [Documentation] Find a basic rpc test service owner and successors and store them if indicated. + ${brt_owner} ${brt_successors}= ClusterManagement.Get_Owner_And_Successors_For_Device ${SERVICE} org.opendaylight.mdsal.ServiceEntityType ${node_to_ask} + BuiltIn.Run_Keyword_If ${store} BuiltIn.Set_Suite_Variable ${brt_owner} ${brt_owner} + BuiltIn.Run_Keyword_If ${store} BuiltIn.Set_Suite_Variable ${brt_successors} ${brt_successors} + BuiltIn.Return_From_Keyword ${brt_owner} ${brt_successors} diff --git a/csit/suites/controller/singleton_service/global_rpc_kill.robot b/csit/suites/controller/singleton_service/global_rpc_kill.robot new file mode 100644 index 0000000000..40a1934b21 --- /dev/null +++ b/csit/suites/controller/singleton_service/global_rpc_kill.robot @@ -0,0 +1,113 @@ +*** Settings *** +Documentation Controller functional HA testing of global singleton rpcs. +... +... Copyright (c) 2017 Cisco Systems, Inc. and others. All rights reserved. +... +... This program and the accompanying materials are made available under the +... terms of the Eclipse Public License v1.0 which accompanies this distribution, +... and is available at http://www.eclipse.org/legal/epl-v10.html +Suite Setup Setup_Suite +Suite Teardown SSHLibrary.Close_All_Connections +Test Setup SetupUtils.Setup_Test_With_Logging_And_Without_Fast_Failing +Test Teardown SetupUtils.Teardown_Test_Show_Bugs_If_Test_Failed +Default Tags critical +Resource ${CURDIR}/../../../libraries/ClusterManagement.robot +Resource ${CURDIR}/../../../libraries/KarafKeywords.robot +Resource ${CURDIR}/../../../libraries/SetupUtils.robot +Resource ${CURDIR}/../../../libraries/TemplatedRequests.robot +Resource ${CURDIR}/../../../libraries/WaitForFailure.robot +Resource ${CURDIR}/../../../variables/Variables.robot + +*** Variables *** +${RPC_URL} /restconf/operations/basic-rpc-test:basic-global +&{EMPTY_DICT} +${SERVICE} Basic-rpc-test'] +${TEST_LOG_LEVEL} info +@{TEST_LOG_COMPONENTS} org.opendaylight.controller.remote.rpc +${EOS_URL} /restconf/operational/entity-owners:entity-owners +@{NO_TAGS} + +*** Test Cases *** +Get_Basic_Rpc_Test_Owner + [Documentation] Find a service owner and successors. + [Tags] @{NO_TAGS} + Get_Present_Brt_Owner_And_Successors 1 store=${True} + +Rpc_Before_Killing_On_Owner + [Documentation] Run rpc on the service owner. + Run_Rpc ${brt_owner} + +Rpc_Before_Kill_On_Successors + [Documentation] Run rpc on non owher cluster nodes. + : FOR ${idx} IN @{brt_successors} + \ Run_Rpc ${idx} + +Kill_Current_Owner_Member + [Documentation] Kill cluster node which is the owner. + [Tags] @{NO_TAGS} + ClusterManagement.Kill_Single_Member ${brt_owner} + BuiltIn.Set Suite variable ${old_brt_owner} ${brt_owner} + BuiltIn.Set Suite variable ${old_brt_successors} ${brt_successors} + +Verify_New_Basic_Rpc_Test_Owner_Elected + [Documentation] Verify new owner of the service is elected. + ${idx}= Collections.Get_From_List ${old_brt_successors} 0 + BuiltIn.Wait_Until_Keyword_Succeeds 5x 2s Verify_Owner_Elected ${True} ${old_brt_owner} ${idx} + Get_Present_Brt_Owner_And_Successors ${idx} store=${True} + +Rpc_On_Remained_Cluster_Nodes + [Documentation] Run rpc on remained cluster nodes. + : FOR ${idx} IN @{old_brt_successors} + \ Run_Rpc ${idx} + +Restart_Killed_Member + [Documentation] Restart killed node + [Tags] @{NO_TAGS} + ClusterManagement.Start_Single_Member ${old_brt_owner} + +Verify_New_Owner_Remained_After_Rejoin + [Documentation] Verify no owner change happened after rejoin. + WaitForFailure.Verify_Keyword_Does_Not_Fail_Within_Timeout 15s 2s Verify_Owner_Elected ${False} ${brt_owner} ${brt_owner} + +Rpc_After_Rejoin_On_New_Owner + [Documentation] Run rpc on the new service owner node. + Run_Rpc ${brt_owner} + +Rpc_After_Rejoin_On_Old_Owner + [Documentation] Run rpc on rejoined cluster node. + Run_Rpc ${old_brt_owner} + +Rpc_After_Rejoin_On_All + [Documentation] Run rpc again on all nodes. + Run_Rpc ${brt_owner} + : FOR ${idx} IN @{brt_successors} + \ Run_Rpc ${idx} + +*** Keywords *** +Setup_Suite + SetupUtils.Setup_Utils_For_Setup_And_Teardown + SetupUtils.Setup_Logging_For_Debug_Purposes_On_List_Or_All ${TEST_LOG_LEVEL} ${TEST_LOG_COMPONENTS} + +Run_Rpc + [Arguments] ${node_idx} + [Documentation] Run rpc and log the entity ownership service details to karaf log. + ... Logging the details was a developer's request during the implementation to improve debugging. + ${session} = Resolve_Http_Session_For_Member member_index=${node_idx} + ${out} = TemplatedRequests.Get_From_Uri ${EOS_URL} session=${session} + KarafKeywords.Log_Message_To_Controller_Karaf EOS rest resp: ${out} + TemplatedRequests.Post_To_Uri ${RPC_URL} ${EMPTY} ${EMPTY_DICT} ${EMPTY_DICT} session=${session} + +Verify_Owner_Elected + [Arguments] ${new_elected} ${old_owner} ${node_to_ask} + [Documentation] Verify new owner was elected or remained the same. + ${owner} ${successors}= Get_Present_Brt_Owner_And_Successors ${node_to_ask} + BuiltIn.Run_Keyword_If ${new_elected} BuiltIn.Should_Not_Be_Equal_As_Numbers ${old_owner} ${owner} + BuiltIn.Run_Keyword_Unless ${new_elected} BuiltIn.Should_Be_Equal_As_numbers ${old_owner} ${owner} + +Get_Present_Brt_Owner_And_Successors + [Arguments] ${node_to_ask} ${store}=${False} + [Documentation] Find a basic rpc test service owner and successors and store them if indicated. + ${brt_owner} ${brt_successors}= ClusterManagement.Get_Owner_And_Successors_For_Device ${SERVICE} org.opendaylight.mdsal.ServiceEntityType ${node_to_ask} + BuiltIn.Run_Keyword_If ${store} BuiltIn.Set_Suite_Variable ${brt_owner} ${brt_owner} + BuiltIn.Run_Keyword_If ${store} BuiltIn.Set_Suite_Variable ${brt_successors} ${brt_successors} + BuiltIn.Return_From_Keyword ${brt_owner} ${brt_successors} diff --git a/csit/testplans/controller-clustering-beryllium.txt b/csit/testplans/controller-clustering-beryllium.txt new file mode 100644 index 0000000000..f09fb07e23 --- /dev/null +++ b/csit/testplans/controller-clustering-beryllium.txt @@ -0,0 +1,8 @@ +# Place the suites in run order: +integration/test/csit/suites/controller/Clustering_Datastore/cluster_ready.robot +integration/test/csit/suites/controller/Clustering_Datastore/carpeople_crud.robot +integration/test/csit/suites/controller/Clustering_Datastore/car_failover_crud.robot +integration/test/csit/suites/controller/Clustering_Datastore/car_persistence_recovery.robot +integration/test/csit/suites/controller/Clustering_Datastore/car_outage_corners.robot +integration/test/csit/suites/controller/Clustering_Datastore/buycar_failover.robot +integration/test/csit/suites/controller/Clustering_Datastore/entity_isolate.robot diff --git a/csit/testplans/controller-clustering-boron.txt b/csit/testplans/controller-clustering-boron.txt new file mode 100644 index 0000000000..f09fb07e23 --- /dev/null +++ b/csit/testplans/controller-clustering-boron.txt @@ -0,0 +1,8 @@ +# Place the suites in run order: +integration/test/csit/suites/controller/Clustering_Datastore/cluster_ready.robot +integration/test/csit/suites/controller/Clustering_Datastore/carpeople_crud.robot +integration/test/csit/suites/controller/Clustering_Datastore/car_failover_crud.robot +integration/test/csit/suites/controller/Clustering_Datastore/car_persistence_recovery.robot +integration/test/csit/suites/controller/Clustering_Datastore/car_outage_corners.robot +integration/test/csit/suites/controller/Clustering_Datastore/buycar_failover.robot +integration/test/csit/suites/controller/Clustering_Datastore/entity_isolate.robot diff --git a/csit/testplans/controller-clustering.txt b/csit/testplans/controller-clustering.txt index f09fb07e23..04bf940874 100644 --- a/csit/testplans/controller-clustering.txt +++ b/csit/testplans/controller-clustering.txt @@ -6,3 +6,5 @@ integration/test/csit/suites/controller/Clustering_Datastore/car_persistence_rec integration/test/csit/suites/controller/Clustering_Datastore/car_outage_corners.robot integration/test/csit/suites/controller/Clustering_Datastore/buycar_failover.robot integration/test/csit/suites/controller/Clustering_Datastore/entity_isolate.robot +integration/test/csit/suites/controller/singleton_service/global_rpc_kill.robot +integration/test/csit/suites/controller/singleton_service/global_rpc_isolate.robot