2 Documentation netconf-connector scaling test suite to find max connected devices
4 ... Copyright (c) 2019 Lumina Networks, Inc. and others. All rights reserved.
6 ... This program and the accompanying materials are made available under the
7 ... terms of the Eclipse Public License v1.0 which accompanies this distribution,
8 ... and is available at http://www.eclipse.org/legal/epl-v10.html
11 ... Increasing numbers of netconf devices will be connected and cleaned up
12 ... while validating and profiling between each iteration.
13 Suite Setup Setup_Everything
14 Suite Teardown Teardown_Everything
15 Test Setup SetupUtils.Setup_Test_With_Logging_And_Without_Fast_Failing
18 Library SSHLibrary timeout=10s
19 Resource ../../../libraries/CheckJVMResource.robot
20 Resource ../../../libraries/KarafKeywords.robot
21 Resource ../../../libraries/NetconfKeywords.robot
22 Resource ../../../libraries/SetupUtils.robot
23 Resource ../../../libraries/SSHKeywords.robot
24 Resource ../../../variables/Variables.robot
27 ${INIT_DEVICE_COUNT} 250
28 ${MAX_DEVICE_COUNT} 5000
29 ${DEVICE_INCREMENT} 250
30 ${DEVICE_NAME_BASE} netconf-scaling-device
31 ${DEVICE_TYPE} full-uri-device
35 ${DEVICES_RESULT_FILE} devices.csv
36 ${INSTALL_TESTTOOL} True
37 ${TESTTOOL_EXECUTABLE} ${EMPTY}
40 Find Max Netconf Devices
41 [Documentation] Find max number of switches starting from ${MIN_SWITCHES} till reaching ${MAX_SWITCHES} in steps of ${STEP_SWITCHES}
42 ${error_message} = BuiltIn.Set Variable Failure initializing suite
43 ${maximum_devices} = BuiltIn.Set Variable ${0}
44 ${discover_time} = BuiltIn.Set Variable 0
45 ${start} = BuiltIn.Convert to Integer ${INIT_DEVICE_COUNT}
46 ${stop} = BuiltIn.Convert to Integer ${MAX_DEVICE_COUNT}
47 ${increment} = BuiltIn.Convert to Integer ${DEVICE_INCREMENT}
48 Run Keyword And Ignore Error CheckJVMResource.Get JVM Memory
49 : FOR ${devices} IN RANGE ${start} ${stop+1} ${increment}
50 \ ${timeout} = BuiltIn.Evaluate ${devices}*${TIMEOUT_FACTOR}
51 \ Log To Console Starting Iteration with ${devices} devices
52 \ Run Keyword If "${INSTALL_TESTTOOL}"=="True" NetconfKeywords.Install_And_Start_Testtool device-count=${devices}
53 \ ... ELSE NetconfKeywords.Start_Testtool ${TESTTOOL_EXECUTABLE} device-count=${devices}
54 \ ${status} ${result} = Run Keyword And Ignore Error NetconfKeywords.Perform_Operation_On_Each_Device Configure_Device timeout=${timeout}
55 \ Exit For Loop If '${status}' == 'FAIL'
56 \ ${status} ${result} = Run Keyword And Ignore Error NetconfKeywords.Perform_Operation_On_Each_Device Wait_Connected timeout=${timeout}
57 \ Exit For Loop If '${status}' == 'FAIL'
58 \ ${status} ${result} = Run Keyword And Ignore Error Issue_Requests_On_Devices ${TOOLS_SYSTEM_IP} ${devices}
60 \ Exit For Loop If '${status}' == 'FAIL'
61 \ ${status} ${result} = Run Keyword And Ignore Error NetconfKeywords.Perform_Operation_On_Each_Device Wait_Connected timeout=${timeout}
62 \ Exit For Loop If '${status}' == 'FAIL'
63 \ ${status} ${result} = Run Keyword And Ignore Error NetconfKeywords.Perform_Operation_On_Each_Device Deconfigure_Device timeout=${timeout}
64 \ Exit For Loop If '${status}' == 'FAIL'
65 \ ${status} ${result} = Run Keyword And Ignore Error NetconfKeywords.Perform_Operation_On_Each_Device Check_Device_Deconfigured timeout=${timeout}
66 \ Exit For Loop If '${status}' == 'FAIL'
67 \ ${maximum_devices} = Set Variable ${devices}
68 \ Run Keyword And Ignore Error CheckJVMResource.Get JVM Memory
69 \ NetconfKeywords.Stop_Testtool
70 [Teardown] Run Keywords NetconfKeywords.Stop_Testtool
71 ... AND Collect_Data_Points ${maximum_devices}
72 ... AND Run Keyword And Ignore Error CheckJVMResource.Get JVM Memory
76 [Arguments] ${devices}
77 [Documentation] Parse and Log relevant information when Scale test finishes
78 OperatingSystem.Append To File ${DEVICES_RESULT_FILE} Max Devices\n
79 OperatingSystem.Append To File ${DEVICES_RESULT_FILE} ${devices}\n
81 Issue_Requests_On_Devices
82 [Arguments] ${client_ip} ${expected_count} ${worker_count}
83 [Documentation] Spawn the specified count of worker threads to issue a GET request to each of the devices.
84 ${current_ssh_connection}= SSHLibrary.Get Connection
85 SSHLibrary.Open_Connection ${client_ip}
86 SSHKeywords.Flexible_Mininet_Login
87 SSHLibrary.Write python getter.py --odladdress=${ODL_SYSTEM_IP} --count=${expected_count} --name=${device_name_base} --workers=${worker_count}
88 : FOR ${number} IN RANGE 1 ${expected_count}+1
89 \ Read_Python_Tool_Operation_Result ${number}
90 SSHLibrary.Read_Until_Prompt
91 SSHLibrary.Close_Connection
92 SSHKeywords.Restore Current SSH Connection From Index ${current_ssh_connection.index}
95 [Documentation] Setup everything needed for the test cases.
96 # Setup resources used by the suite.
97 RequestsLibrary.Create_Session operational http://${ODL_SYSTEM_IP}:${RESTCONFPORT}${OPERATIONAL_API} auth=${AUTH}
98 SSHLibrary.Set_Default_Configuration prompt=${TOOLS_SYSTEM_PROMPT}
99 SetupUtils.Setup_Utils_For_Setup_And_Teardown
100 NetconfKeywords.Setup_Netconf_Keywords
101 # Deploy testing tools.
102 SSHLibrary.Put_File ${CURDIR}/../../../../tools/netconf_tools/getter.py
103 SSHLibrary.Put_File ${CURDIR}/../../../libraries/AuthStandalone.py
104 ${device_type}= BuiltIn.Set_Variable_If ${USE_NETCONF_CONNECTOR}==${True} default ${device_type}
105 BuiltIn.Set_Suite_Variable ${device_type}
108 [Documentation] Teardown the test infrastructure, perform cleanup and release all resources.
109 RequestsLibrary.Delete_All_Sessions
110 NetconfKeywords.Stop_Testtool
113 [Arguments] ${current_name}
114 [Documentation] Operation for configuring the device.
115 KarafKeywords.Log_Message_To_Controller_Karaf Configuring device ${current_name} to Netconf
116 NetconfKeywords.Configure_Device_In_Netconf ${current_name} device_type=${device_type} device_port=${current_port}
117 KarafKeywords.Log_Message_To_Controller_Karaf Device ${current_name} configured
120 [Arguments] ${current_name}
121 [Documentation] Operation for waiting until the device is connected.
122 KarafKeywords.Log_Message_To_Controller_Karaf Waiting for device ${current_name} to connect
123 NetconfKeywords.Wait_Device_Connected ${current_name} period=0.5s timeout=120s
124 KarafKeywords.Log_Message_To_Controller_Karaf Device ${current_name} connected
126 Read_Python_Tool_Operation_Result
127 [Arguments] ${number}
128 [Documentation] Read and process a report line emitted from the Python tool that corresponds to the device with the given number.
129 ${test}= SSHLibrary.Read_Until_Regexp \\n
130 ${test}= String.Split_String ${test} |
131 ${response}= Collections.Get_From_List ${test} 0
132 ${message}= Collections.Get_From_List ${test} 1
133 BuiltIn.Run_Keyword_If '${response}' == 'ERROR' Fail Error getting data: ${message}
134 ${start}= Collections.Get_From_List ${test} 1
135 ${stop}= Collections.Get_From_List ${test} 2
136 ${ellapsed}= Collections.Get_From_List ${test} 3
137 BuiltIn.Log DATA REQUEST RESULT: Device=${number} StartTime=${start} StopTime=${stop} EllapsedTime=${ellapsed}
138 ${data}= Collections.Get_From_List ${test} 4
139 ${expected}= BuiltIn.Set_Variable '<data xmlns="${ODL_NETCONF_NAMESPACE}"></data>'
140 BuiltIn.Should_Be_Equal_As_Strings ${data} ${expected}
143 [Arguments] ${current_name}
144 [Documentation] Operation for deconfiguring the device.
145 KarafKeywords.Log_Message_To_Controller_Karaf Deconfiguring device ${current_name}
146 NetconfKeywords.Remove_Device_From_Netconf ${current_name}
147 KarafKeywords.Log_Message_To_Controller_Karaf Device ${current_name} deconfigured
149 Check_Device_Deconfigured
150 [Arguments] ${current_name}
151 [Documentation] Operation for making sure the device is really deconfigured.
152 KarafKeywords.Log_Message_To_Controller_Karaf Waiting for device ${current_name} to disappear
153 NetconfKeywords.Wait_Device_Fully_Removed ${current_name} period=0.5s timeout=120s
154 KarafKeywords.Log_Message_To_Controller_Karaf Device ${current_name} removed