Fixed too tight timeouts for deconfiguration

Timeouts of 10 seconds for deconfiguring the device is too
tight when there are other scaling suites executed before
and the memory leak bug 4514 is present. The needed timeout
is 120 seconds as I had seen 20 to 30 second garbage
collection pauses on just a 1GB heap usage and the bug
causes the ODL to leak about 5GB if two scaling suites
are present.

There are two timeouts, one is the Karaf console timeout
and the other one is the timeout waiting for the device
to disappear after it was deconfigured. The large timeout
is necessary in the second case because the "device
disappeared" check is executed immediately after the request
to remove it from the configuration was sent.

To avoid "test running for 2 weeks if something bad happens"
type of situation a global timeout logic was also added. The
default timeout is 30 minutes but the scaling suite is
giving less time to the loops.

Change-Id: Ie59a2f9d7a6ee9999f202ba209caf03efd05a801
Signed-off-by: Jozef Behran <jbehran@cisco.com>

diff --git a/csit/libraries/NetconfKeywords.robot b/csit/libraries/NetconfKeywords.robot
index 2e5e8d034c7e07387bae066ceb64647a3ae9a44b..43af31194c6770bec5177fb366b1771acb527807 100644 (file)
--- a/csit/libraries/NetconfKeywords.robot
+++ b/csit/libraries/NetconfKeywords.robot
@@ -12,6 +12,7 @@ Documentation     Perform complex operations on netconf.
 ...               netconf operations into reusable keywords to make writing netconf
 ...               test suites easier.
 Library           Collections
+Library           DateTime
 Library           RequestsLibrary
 Resource          NetconfViaRestconf.robot
 Resource          SSHKeywords.robot
@@ -166,7 +167,10 @@ Stop_Testtool
     SSHLibrary.Get_File    testtool.log
 
 NetconfKeywords__Perform_Operation_With_Checking_On_Next_Device
-    [Arguments]    ${operation}
+    [Arguments]    ${operation}    ${deadline_Date}
+    ${current_Date}=    DateTime.Get_Current_Date
+    ${ellapsed_seconds}=    DateTime.Subtract_Date_From_Date    ${deadline_Date}    ${current_Date}
+    BuiltIn.Run_Keyword_If    ${ellapsed_seconds}<0    Fail    The global time out period expired
     ${number}=    BuiltIn.Evaluate    ${current_port}-${BASE_NETCONF_DEVICE_PORT}+1
     BuiltIn.Wait_Until_Keyword_Succeeds    10s    1s    NetconfKeywords.Check_Device_Up_And_Running    ${number}
     BuiltIn.Run_Keyword    ${operation}    ${DEVICE_NAME_BASE}-${number}
@@ -174,6 +178,8 @@ NetconfKeywords__Perform_Operation_With_Checking_On_Next_Device
     BuiltIn.Set_Suite_Variable    ${current_port}    ${next}
 
 Perform_Operation_On_Each_Device
-    [Arguments]    ${operation}    ${count}=${DEVICE_COUNT}
+    [Arguments]    ${operation}    ${count}=${DEVICE_COUNT}    ${timeout}=30m
+    ${current_Date}=    DateTime.Get_Current_Date
+    ${deadline_Date}=    DateTime.Add_Time_To_Date    ${current_Date}    ${timeout}
     BuiltIn.Set_Suite_Variable    ${current_port}    ${BASE_NETCONF_DEVICE_PORT}
-    BuiltIn.Repeat_Keyword    ${count} times    NetconfKeywords__Perform_Operation_With_Checking_On_Next_Device    ${operation}
+    BuiltIn.Repeat_Keyword    ${count} times    NetconfKeywords__Perform_Operation_With_Checking_On_Next_Device    ${operation}    ${deadline_Date}

diff --git a/csit/suites/netconf/scale/getsingle.robot b/csit/suites/netconf/scale/getsingle.robot
index 7fac02ac2e132280cfbee00bcd8cb6b34b68633e..9ef0e3de77f7a967260bea820099c4ec87cff44c 100644 (file)
--- a/csit/suites/netconf/scale/getsingle.robot
+++ b/csit/suites/netconf/scale/getsingle.robot
@@ -29,16 +29,19 @@ ${DEVICE_COUNT}    500
 Configure_Devices_Onto_Netconf
     [Documentation]    Make requests to configure the testtool devices.
     [Tags]    critical
-    NetconfKeywords.Perform_Operation_On_Each_Device    Configure_Device
+    ${timeout}=    BuiltIn.Evaluate    ${DEVICE_COUNT}*10
+    NetconfKeywords.Perform_Operation_On_Each_Device    Configure_Device    timeout=${timeout}
 
 Get_Data_From_Devices
     [Documentation]    Ask testtool devices for data.
-    NetconfKeywords.Perform_Operation_On_Each_Device    Check_Device_Data
+    ${timeout}=    BuiltIn.Evaluate    ${DEVICE_COUNT}*2
+    NetconfKeywords.Perform_Operation_On_Each_Device    Check_Device_Data    timeout=${timeout}
 
 Deconfigure_Devices_From_Netconf
     [Documentation]    Make requests to deconfigure the testtool devices.
     [Tags]    critical
-    NetconfKeywords.Perform_Operation_On_Each_Device    Deconfigure_Device
+    ${timeout}=    BuiltIn.Evaluate    ${DEVICE_COUNT}*10
+    NetconfKeywords.Perform_Operation_On_Each_Device    Deconfigure_Device    timeout=${timeout}
     [Teardown]    Report_Failure_Due_To_Bug    4547
 
 *** Keywords ***
@@ -50,6 +53,7 @@ Setup_Everything
     SSHLibrary.Set_Default_Configuration    prompt=${TOOLS_SYSTEM_PROMPT}
     SetupUtils.Setup_Utils_For_Setup_And_Teardown
     NetconfKeywords.Setup_Netconf_Keywords
+    KarafKeywords.Configure_Timeout_For_Karaf_Console    120s
     # Connect to the tools machine
     SSHLibrary.Open_Connection    ${TOOLS_SYSTEM_IP}
     Utils.Flexible_Mininet_Login
@@ -85,5 +89,5 @@ Deconfigure_Device
     KarafKeywords.Log_Message_To_Controller_Karaf    Removing device ${current_name}
     NetconfKeywords.Remove_Device_From_Netconf    ${current_name}
     KarafKeywords.Log_Message_To_Controller_Karaf    Waiting for device ${current_name} to disappear
-    NetconfKeywords.Wait_Device_Fully_Removed    ${current_name}    period=0.5s
+    NetconfKeywords.Wait_Device_Fully_Removed    ${current_name}    period=0.5s    timeout=120s
     KarafKeywords.Log_Message_To_Controller_Karaf    Device ${current_name} removed