Put testtool startup into its own testcase

The Install_And_Start_Testtool contains an attempt to wait
for testtool to become usable. This is done by waiting for
the first test tool device to come online. This was done with
assumption that further testtool devices will come online
very quickly after the first one.

However it turns out the test tool startup may take
considerable amount of time even after this first device
comes online. Thus the "the remaining testtool devices come
online quickly" assumption is not valid and overlapping this
time consuming operation with the device configuration test
case in the scaling test suite causes the test case to be
blocked waiting for testtool to supply the next device,
destroying the validity of its results.

To fix this amend the "wait for testtool to boot" action
to make sure that all testtool devices are online and then
move the testtool startup into the first test case in the
suite as it now can fail.

Change-Id: Ie883d375e95e51d78c8343133182c644bde6e1df
Signed-off-by: Jozef Behran <jbehran@cisco.com>

diff --git a/csit/libraries/NetconfKeywords.robot b/csit/libraries/NetconfKeywords.robot
index 74f6b8cda962d96b844f2dc17118cdd3989be294..5c1237084f811afe0f3ff689dfe6c09953ac3617 100644 (file)
--- a/csit/libraries/NetconfKeywords.robot
+++ b/csit/libraries/NetconfKeywords.robot
@@ -24,6 +24,7 @@ ${DIRECTORY_WITH_DEVICE_TEMPLATES}    ${CURDIR}/../variables/netconf/device
 ${FIRST_TESTTOOL_PORT}    17830
 ${BASE_NETCONF_DEVICE_PORT}    17830
 ${DEVICE_NAME_BASE}    netconf-scaling-device
+${TESTTOOL_DEVICE_TIMEOUT}    60s
 
 *** Keywords ***
 Setup_NetconfKeywords
@@ -125,6 +126,11 @@ NetconfKeywords__Check_Device_Is_Up
     ${count}=    SSHKeywords.Count_Port_Occurences    ${last-port}    LISTEN    java
     BuiltIn.Should_Be_Equal_As_Integers    ${count}    1
 
+NetconfKeywords__Wait_Device_Is_Up_And_Running
+    [Arguments]    ${device_name}
+    ${number}=    BuiltIn.Evaluate    '${device_name}'.split('-').pop()
+    BuiltIn.Wait_Until_Keyword_Succeeds    ${TESTTOOL_DEVICE_TIMEOUT}    1s    Check_Device_Up_And_Running    ${number}
+
 Install_And_Start_Testtool
     [Arguments]    ${device-count}=10    ${debug}=true    ${schemas}=none    ${tool_options}=${EMPTY}    ${java_options}=${TESTTOOL_DEFAULT_JAVA_OPTIONS}
     [Documentation]    Install and run testtool. Also arrange to collect its output into a log file.
@@ -139,11 +145,10 @@ Install_And_Start_Testtool
     ${command}    BuiltIn.Set_Variable    java ${java_options} -jar ${filename} ${tool_options} --device-count ${device-count} --debug ${debug} ${schemas_option}
     BuiltIn.Log    Running testtool: ${command}
     SSHLibrary.Write    ${command} >testtool.log 2>&1
-    # Wait for the testtool to boot up.
-    ${timeout}=    BuiltIn.Evaluate    (${device-count}/3)+5
-    BuiltIn.Wait_Until_Keyword_Succeeds    ${timeout}s    1s    NetconfKeywords__Check_Device_Is_Up    ${FIRST_TESTTOOL_PORT}
     # Store information needed by other keywords.
     BuiltIn.Set_Suite_Variable    ${NetconfKeywords__testtool_device_count}    ${device-count}
+    # Wait for the testtool to boot up.
+    Perform_Operation_On_Each_Device    NetconfKeywords__Wait_Device_Is_Up_And_Running
 
 Check_Device_Up_And_Running
     [Arguments]    ${device-number}
@@ -167,7 +172,6 @@ NetconfKeywords__Perform_Operation_With_Checking_On_Next_Device
     ${ellapsed_seconds}=    DateTime.Subtract_Date_From_Date    ${deadline_Date}    ${current_Date}
     BuiltIn.Run_Keyword_If    ${ellapsed_seconds}<0    Fail    The global time out period expired
     ${number}=    BuiltIn.Evaluate    ${current_port}-${BASE_NETCONF_DEVICE_PORT}+1
-    BuiltIn.Wait_Until_Keyword_Succeeds    10s    1s    NetconfKeywords.Check_Device_Up_And_Running    ${number}
     BuiltIn.Run_Keyword    ${operation}    ${DEVICE_NAME_BASE}-${number}
     ${next}=    BuiltIn.Evaluate    ${current_port}+1
     BuiltIn.Set_Suite_Variable    ${current_port}    ${next}

diff --git a/csit/suites/netconf/CRUD/CRUD.robot b/csit/suites/netconf/CRUD/CRUD.robot
index ed14c3a6bc5b31d97bd765d0cd58acc977ebba97..b0140ce696eb46032d801aca8fe0968d93f791c4 100644 (file)
--- a/csit/suites/netconf/CRUD/CRUD.robot
+++ b/csit/suites/netconf/CRUD/CRUD.robot
@@ -32,6 +32,10 @@ ${DIRECTORY_WITH_TEMPLATE_FOLDERS}    ${CURDIR}/../../../variables/netconf/CRUD
 ${device_name}    netconf-test-device
 
 *** Test Cases ***
+Start_Testtool
+    [Documentation]    Deploy and start test tool, then wait for all its devices to become online.
+    NetconfKeywords.Install_And_Start_Testtool    device-count=1    schemas=${CURDIR}/../../../variables/netconf/CRUD/schemas
+
 Check_Device_Is_Not_Configured_At_Beginning
     [Documentation]    Sanity check making sure our device is not there. Fail if found.
     [Tags]    critical
@@ -111,7 +115,6 @@ Setup_Everything
     # Connect to the Mininet machine
     SSHLibrary.Open_Connection    ${TOOLS_SYSTEM_IP}    prompt=${TOOLS_SYSTEM_PROMPT}
     Utils.Flexible_Mininet_Login
-    NetconfKeywords.Install_And_Start_Testtool    device-count=10    schemas=${CURDIR}/../../../variables/netconf/CRUD/schemas
 
 Teardown_Everything
     [Documentation]    Teardown the test infrastructure, perform cleanup and release all resources.

diff --git a/csit/suites/netconf/scale/getmulti.robot b/csit/suites/netconf/scale/getmulti.robot
index 74d4042ad7d1fe5b3fc12c2973796a7102351fdd..ef63e4262ac346d7d01a15d28e71ca3d2f24428e 100644 (file)
--- a/csit/suites/netconf/scale/getmulti.robot
+++ b/csit/suites/netconf/scale/getmulti.robot
@@ -32,6 +32,10 @@ ${device_name_base}    netconf-scaling-device
 ${base_port}      17830
 
 *** Test Cases ***
+Start_Test_Tool
+    [Documentation]    Deploy and start test tool, then wait for all its devices to become online.
+    NetconfKeywords.Install_And_Start_Testtool    device-count=${DEVICE_COUNT}
+
 Configure_Devices_On_Netconf
     [Documentation]    Make requests to configure the testtool devices.
     ${timeout}=    BuiltIn.Evaluate    ${DEVICE_COUNT}*10
@@ -76,8 +80,7 @@ Setup_Everything
     # Connect to the tools machine
     SSHLibrary.Open_Connection    ${TOOLS_SYSTEM_IP}
     Utils.Flexible_Mininet_Login
-    # Deploy testtool on it
-    NetconfKeywords.Install_And_Start_Testtool    device-count=${DEVICE_COUNT}
+    # Deploy testing tools on it.
     SSHLibrary.Put_File    ${CURDIR}/../../../../tools/netconf_tools/getter.py
     SSHLibrary.Put_File    ${CURDIR}/../../../libraries/AuthStandalone.py
 

diff --git a/csit/suites/netconf/scale/getsingle.robot b/csit/suites/netconf/scale/getsingle.robot
index 9ef0e3de77f7a967260bea820099c4ec87cff44c..8931c9bc2fc5097e851e9bb35c4b49535e0f05ac 100644 (file)
--- a/csit/suites/netconf/scale/getsingle.robot
+++ b/csit/suites/netconf/scale/getsingle.robot
@@ -26,6 +26,10 @@ Variables         ${CURDIR}/../../../variables/Variables.py
 ${DEVICE_COUNT}    500
 
 *** Test Cases ***
+Start_Test_Tool
+    [Documentation]    Deploy and start test tool, then wait for all its devices to become online.
+    NetconfKeywords.Install_And_Start_Testtool    device-count=${DEVICE_COUNT}
+
 Configure_Devices_Onto_Netconf
     [Documentation]    Make requests to configure the testtool devices.
     [Tags]    critical