csit/suites/netconf/clusteringscale/performance.robot

   1 *** Settings ***
   2 Documentation     netconf-restperfclient update performance test suite (clustered setup).
   3 ...
   4 ...               Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.
   5 ...
   6 ...               This program and the accompanying materials are made available under the
   7 ...               terms of the Eclipse Public License v1.0 which accompanies this distribution,
   8 ...               and is available at http://www.eclipse.org/legal/epl-v10.html
   9 ...
  10 ...
  11 ...               Perform given count of update operations on device data mounted onto a
  12 ...               netconf connector (using the netconf-testtool-restperfclient tool) and
  13 ...               see how much time it took. More exactly, it sends the data to a restconf
  14 ...               mountpoint of the netconf connector belonging to the device, which turns
  15 ...               out to turn the first request sent to a "create" request and the
  16 ...               remaining requests to "update" requests (due to how the testtool device
  17 ...               behavior is implemented).
  18 ...
  19 ...               The difference from the "single node" test suite (see
  20 ...               ../scale/performance.robot) is that the device is configured and the data
  21 ...               on it created using one node in the cluster and the update operations are
  22 ...               issued on a different node. This forces the cluster nodes to communicate
  23 ...               with each other about the data to be sent to the device.
  24 Suite Setup       Setup_Everything
  25 Suite Teardown    Teardown_Everything
  26 Test Setup        SetupUtils.Setup_Test_With_Logging_And_Fast_Failing
  27 Test Teardown     SetupUtils.Teardown_Test_Show_Bugs_And_Start_Fast_Failing_If_Test_Failed
  28 Default Tags      1node    netconf    critical
  29 Library           Collections
  30 Library           RequestsLibrary
  31 Library           OperatingSystem
  32 Library           String
  33 Library           SSHLibrary    timeout=10s
  34 Resource          ${CURDIR}/../../../libraries/KarafKeywords.robot
  35 Resource          ${CURDIR}/../../../libraries/NetconfKeywords.robot
  36 Resource          ${CURDIR}/../../../libraries/NexusKeywords.robot
  37 Resource          ${CURDIR}/../../../libraries/RestPerfClient.robot
  38 Resource          ${CURDIR}/../../../libraries/SetupUtils.robot
  39 Resource          ${CURDIR}/../../../libraries/Utils.robot
  40 Variables         ${CURDIR}/../../../variables/Variables.py
  41
  42 *** Variables ***
  43 ${DEVICE_NAME}    ${FIRST_TESTTOOL_PORT}-sim-device
  44 ${REQUEST_COUNT}    16384
  45 ${directory_with_crud_templates}    ${CURDIR}/../../../variables/netconf/CRUD
  46 ${DEVICE_DATA_CONNECT_TIMEOUT}    60s
  47
  48 *** Test Cases ***
  49 Start_Testtool
  50     [Documentation]    Deploy and start test tool, then wait for all its devices to become online.
  51     # Start test tool
  52     SSHLibrary.Switch_Connection    ${testtool}
  53     NetconfKeywords.Install_And_Start_Testtool    device-count=1    debug=false    schemas=${CURDIR}/../../../variables/netconf/CRUD/schemas
  54
  55 Configure_Device_On_Netconf
  56     [Documentation]    Configure the testtool device on Netconf connector, using node 1.
  57     NetconfKeywords.Configure_Device_In_Netconf    ${DEVICE_NAME}    device_type=configure-via-topology    session=node1
  58
  59 Wait_For_Device_To_Become_Connected
  60     [Documentation]    Wait until the device becomes available through Netconf on node 1.
  61     NetconfKeywords.Wait_Device_Connected    ${DEVICE_NAME}    session=node1
  62
  63 Wait_For_Device_Data_To_Be_Seen
  64     [Documentation]    Wait until the device data show up at node 2.
  65     BuiltIn.Wait_Until_Keyword_Succeeds    ${DEVICE_DATA_CONNECT_TIMEOUT}    1s    Check_Data_Present
  66
  67 Create_Device_Data
  68     [Documentation]    Send some sample test data into the device through node 2 and check that the request went OK.
  69     ${template_as_string}=    BuiltIn.Create_Dictionary    DEVICE_NAME=${device_name}
  70     TemplatedRequests.Post_As_Xml_Templated    ${directory_with_crud_templates}${/}cars    ${template_as_string}    session=node2
  71
  72 Run_Restperfclient
  73     [Documentation]    Deploy and execute restperfclient, asking it to send the specified amount of requests to the netconf connector of the device through node 3.
  74     ...    The duration of this test case is the main performance metric.
  75     ${url}=    BuiltIn.Set_Variable    /rests/data/network-topology:network-topology/topology\=topology-netconf/node\=${DEVICE_NAME}/yang-ext:mount/car:cars
  76     RestPerfClient.Invoke_Restperfclient    ${TESTTOOL_DEVICE_TIMEOUT}    ${url}    ip=${ODL_SYSTEM_3_IP}
  77
  78 Check_For_Failed_Requests
  79     [Documentation]    Make sure there are no failed requests in the restperfclient log.
  80     ...    This is a separate test case to distinguish between restperfclient
  81     ...    failure and failed requests. Failed requests are rejected because
  82     ...    we don't want to test performance of ODL rejecting our requests.
  83     ...    If this test case fails, then the duration of Run_Restperfclient
  84     ...    cannot be trusted to show the real performance of the cluster.
  85     ${result}=    RestPerfClient.Grep_Restperfclient_Log    thread timed out
  86     BuiltIn.Should_Be_Equal    '${result}'    ''
  87     ${result}=    RestPerfClient.Grep_Restperfclient_Log    Request failed
  88     BuiltIn.Should_Be_Equal    '${result}'    ''
  89     ${result}=    RestPerfClient.Grep_Restperfclient_Log    Status code
  90     BuiltIn.Should_Be_Equal    '${result}'    ''
  91
  92 Cleanup_And_Collect
  93     [Documentation]    Deconfigure the testtool device on Netconf connector using node 1.
  94     [Setup]    SetupUtils.Setup_Test_With_Logging_And_Without_Fast_Failing
  95     RestPerfClient.Collect_From_Restperfclient
  96     NetconfKeywords.Remove_Device_From_Netconf    ${DEVICE_NAME}    session=node1
  97
  98 *** Keywords ***
  99 Setup_Everything
 100     [Documentation]    Setup everything needed for the test cases.
 101     # Calculate and set the value of the timeout
 102     ${value}=    BuiltIn.Evaluate    ${REQUEST_COUNT}/10+20
 103     Utils.Set_User_Configurable_Variable_Default    TESTTOOL_DEVICE_TIMEOUT    ${value} s
 104     # Setup resources used by the suite.
 105     SetupUtils.Setup_Utils_For_Setup_And_Teardown
 106     NetconfKeywords.Setup_Netconf_Keywords
 107     RestPerfClient.Setup_Restperfclient
 108     # Connect to the tools system (testtool)
 109     ${testtool}=    SSHKeywords.Open_Connection_To_Tools_System
 110     BuiltIn.Set_Suite_Variable    ${testtool}    ${testtool}
 111     # Create sessions
 112     RequestsLibrary.Create_Session    node1    http://${ODL_SYSTEM_1_IP}:${RESTCONFPORT}    headers=${HEADERS_XML}    auth=${AUTH}
 113     RequestsLibrary.Create_Session    node2    http://${ODL_SYSTEM_2_IP}:${RESTCONFPORT}    headers=${HEADERS_XML}    auth=${AUTH}
 114
 115 Check_Data_Present
 116     ${url}=    Builtin.Set_Variable    ${REST_API}/network-topology:network-topology/topology=topology-netconf/node=${DEVICE_NAME}/yang-ext:mount?content=config
 117     ${data}=    TemplatedRequests.Get_As_Xml_From_Uri    ${url}    session=node2
 118     BuiltIn.Should_Be_Equal_As_Strings    ${data}    <data xmlns="${ODL_NETCONF_NAMESPACE}"></data>
 119
 120 Teardown_Everything
 121     [Documentation]    Teardown the test infrastructure, perform cleanup and release all resources.
 122     RequestsLibrary.Delete_All_Sessions
 123     RestPerfClient.Teardown_Restperfclient
 124     SSHLibrary.Switch_Connection    ${testtool}
 125     BuiltIn.Run_Keyword_And_Ignore_Error    NetconfKeywords.Stop_Testtool