Add variable to select configure netconf device method
[integration/test.git] / csit / suites / netconf / restperfclient / mdsal.robot
1 *** Settings ***
2 Documentation     netconf-restperfclient MDSAL performance test suite.
3 ...
4 ...               Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.
5 ...
6 ...               This program and the accompanying materials are made available under the
7 ...               terms of the Eclipse Public License v1.0 which accompanies this distribution,
8 ...               and is available at http://www.eclipse.org/legal/epl-v10.html
9 ...
10 ...
11 ...               Perform given count of update operations on ODL MDSAL. In first half the
12 ...               requests are directed directly to MDSAL via Restconf and in the second
13 ...               half the MDSAL is mounted onto a netconf connector and the reqursts are
14 ...               directed to that connector. In both cases the netconf-testtool-restperfclient
15 ...               tool is used to generate and send the requests and the requests are sent
16 ...               synchronously as the netconf connector mounted MDSAL does not support
17 ...               asynchronous requests. The restperfclient is used to generate the "update"
18 ...               requests, the "create" request is issued in a sepate test case.
19 Suite Setup       Setup_Everything
20 Suite Teardown    Teardown_Everything
21 Test Setup        SetupUtils.Setup_Test_With_Logging_And_Fast_Failing
22 Test Teardown     SetupUtils.Teardown_Test_Show_Bugs_And_Start_Fast_Failing_If_Test_Failed
23 Library           RequestsLibrary
24 Library           OperatingSystem
25 Library           SSHLibrary    timeout=10s
26 Resource          ${CURDIR}/../../../libraries/NetconfKeywords.robot
27 Resource          ${CURDIR}/../../../libraries/NexusKeywords.robot
28 Resource          ${CURDIR}/../../../libraries/RestPerfClient.robot
29 Resource          ${CURDIR}/../../../libraries/SetupUtils.robot
30 Resource          ${CURDIR}/../../../libraries/TemplatedRequests.robot
31 Resource          ${CURDIR}/../../../libraries/Utils.robot
32 Variables         ${CURDIR}/../../../variables/Variables.py
33
34 *** Variables ***
35 ${DIRECTORY_WITH_TEMPLATE_FOLDERS}    ${CURDIR}/../../../variables/netconf/RestPerfClient
36 ${REQUEST_COUNT}    65536
37 ${device_type}    default
38
39 *** Test Cases ***
40 Create_Test_Data_For_Direct_Access
41     [Documentation]    Send some sample test data into the device and check that the request went OK.
42     TemplatedRequests.Post_As_Xml_Templated    ${DIRECTORY_WITH_TEMPLATE_FOLDERS}${/}cars    {}
43
44 Run_RestPerfClient_Directly_On_MDSAL
45     [Documentation]    Deploy and execute restperfclient, asking it to send the specified amount of requests to the MDSAL via Restconf.
46     ${url}=    BuiltIn.Set_Variable    /restconf/config/car:cars
47     RestPerfClient.Invoke_Restperfclient    ${DIRECT_MDSAL_TIMEOUT}    ${url}    testcase=direct
48
49 Check_For_Failed_Direct_MDSAL_Requests
50     [Documentation]    Make sure there are no failed requests in the restperfclient log.
51     ...    This is a separate test case to distinguish between restperfclient
52     ...    failure and failed requests. Failed requests are rejected because
53     ...    we don't want to test performance of ODL rejecting our requests.
54     [Setup]    SetupUtils.Setup_Test_With_Logging_And_Without_Fast_Failing
55     ${result}=    RestPerfClient.Grep_Restperfclient_Log    thread timed out
56     BuiltIn.Should_Be_Equal    '${result}'    ''
57     ${result}=    RestPerfClient.Grep_Restperfclient_Log    Request failed
58     BuiltIn.Should_Be_Equal    '${result}'    ''
59     ${result}=    RestPerfClient.Grep_Restperfclient_Log    Status code
60     BuiltIn.Should_Be_Equal    '${result}'    ''
61
62 Cleanup_And_Collect_For_Direct_Access
63     [Documentation]    Cleanup the test data produced by the direct MDSAL access.
64     [Setup]    SetupUtils.Setup_Test_With_Logging_And_Without_Fast_Failing
65     RestPerfClient.Collect_From_Restperfclient
66     TemplatedRequests.Delete_Templated    ${DIRECTORY_WITH_TEMPLATE_FOLDERS}${/}cars-delete    {}
67
68 Create_Test_Data_For_Connector_Access
69     [Documentation]    Create the test data container again so it is ready for the netconf connector test.
70     [Setup]    SetupUtils.Setup_Test_With_Logging_And_Without_Fast_Failing
71     TemplatedRequests.Post_As_Xml_Templated    ${DIRECTORY_WITH_TEMPLATE_FOLDERS}${/}cars    {}
72
73 Configure_ODL_As_A_Device_On_Netconf
74     [Documentation]    Configure ODL MDSAL Northbound as a Netconf device on a Netconf connector.
75     NetconfKeywords.Configure_Device_In_Netconf    odl-mdsal-northbound-via-netconf-connector    device_type=${device_type}    device_address=${ODL_SYSTEM_IP}    device_port=${ODL_NETCONF_MDSAL_PORT}    device_user=${ODL_NETCONF_USER}    device_password=${ODL_NETCONF_PASSWORD}
76     NetconfKeywords.Wait_Device_Connected    odl-mdsal-northbound-via-netconf-connector
77
78 Run_RestPerfClient_Through_Netconf_Connector
79     [Documentation]    Ask RestPerfClient to send the requests to the MDSAL mapped via a netconf connector.
80     ${url}=    BuiltIn.Set_Variable    /restconf/config/network-topology:network-topology/topology/topology-netconf/node/odl-mdsal-northbound-via-netconf-connector/yang-ext:mount/car:cars
81     RestPerfClient.Invoke_Restperfclient    ${NETCONF_CONNECTOR_MDSAL_TIMEOUT}    ${url}    testcase=netconf-connector
82
83 Check_For_Failed_Netconf_Connector_Requests
84     [Documentation]    Make sure there are no failed requests in the restperfclient log.
85     ...    This is a separate test case to distinguish between restperfclient
86     ...    failure and failed requests. Failed requests are rejected because
87     ...    we don't want to test performance of ODL rejecting our requests.
88     [Setup]    SetupUtils.Setup_Test_With_Logging_And_Without_Fast_Failing
89     Set_Known_Bug_Id    5581
90     ${result}=    RestPerfClient.Grep_Restperfclient_Log    thread timed out
91     BuiltIn.Should_Be_Equal    '${result}'    ''
92     Set_Unknown_Bug_Id
93     ${result}=    RestPerfClient.Grep_Restperfclient_Log    Request failed
94     BuiltIn.Should_Be_Equal    '${result}'    ''
95     ${result}=    RestPerfClient.Grep_Restperfclient_Log    Status code
96     BuiltIn.Should_Be_Equal    '${result}'    ''
97
98 Deconfigure_ODL_From_Netconf
99     [Documentation]    Deconfigure the ODL MDSAL Northbound attached to a Netconf connector.
100     [Setup]    SetupUtils.Setup_Test_With_Logging_And_Without_Fast_Failing
101     NetconfKeywords.Remove_Device_From_Netconf    odl-mdsal-northbound-via-netconf-connector
102
103 Cleanup_And_Collect_For_Connector_Access
104     [Documentation]    Delete the test data produced by the Netconf connector MDSAL access.
105     [Setup]    SetupUtils.Setup_Test_With_Logging_And_Without_Fast_Failing
106     RestPerfClient.Collect_From_Restperfclient
107     TemplatedRequests.Delete_Templated    ${DIRECTORY_WITH_TEMPLATE_FOLDERS}${/}cars-delete    {}
108
109 *** Keywords ***
110 Setup_Everything
111     [Documentation]    Setup everything needed for the test cases.
112     # Setup resources used by the suite.
113     SetupUtils.Setup_Utils_For_Setup_And_Teardown
114     NetconfKeywords.Setup_Netconf_Keywords
115     RestPerfClient.Setup_Restperfclient
116     # Calculate timeouts
117     ${value}=    BuiltIn.Evaluate    ${REQUEST_COUNT}/50+10
118     Utils.Set_User_Configurable_Variable_Default    DIRECT_MDSAL_TIMEOUT    ${value} s
119     ${value}=    BuiltIn.Evaluate    ${REQUEST_COUNT}/10+10
120     Utils.Set_User_Configurable_Variable_Default    NETCONF_CONNECTOR_MDSAL_TIMEOUT    ${value} s
121
122 Teardown_Everything
123     [Documentation]    Teardown the test infrastructure, perform cleanup and release all resources.
124     RequestsLibrary.Delete_All_Sessions
125     RestPerfClient.Teardown_Restperfclient