Dump karaf threads if karaf does not startup

Debug for karaf not starting up, very intermittent issue.

Look to see if port 2550 is in LISTEN mode, within 60 secs
after starting karaf. If not, dump threads.

JIRA: CONTROLLER-1845

Change-Id: I626506f5317a087d4d9411e14c77cfe996dbd520
Signed-off-by: Victor Pickard <vpickard@redhat.com>

diff --git a/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot b/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot
index 2254b76bceeb7f18b6a0f9f5d75e7fa92dfb802d..d8574f53f7dc827ee85acb4c3a0b98777daf79dc 100644 (file)
--- a/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot
+++ b/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot
@@ -35,5 +35,20 @@ Unset_Tell_Based_Protocol_Usage
 Start_All_And_Sync
     [Documentation]    Start each member and wait for sync.
     ClusterManagement.Start_Members_From_List_Or_All
-    BuiltIn.Wait_Until_Keyword_Succeeds    300s    10s    ShardStability.Shards_Stability_Get_Details    ${DEFAULT_SHARD_LIST}    verify_restconf=True
-    ClusterManagement.Run_Bash_Command_On_List_Or_All    ps -ef | grep java
+    BuiltIn.Wait_Until_Keyword_Succeeds    60s    10s    ClusterManagement.Run_Bash_Command_On_List_Or_All    netstat -punta
+    ${index_list} =    List_Indices_Or_All
+    : FOR    ${index}    IN    @{index_list}
+    \    ${output} =    ClusterManagement.Check_Bash_Command_On_Member    command=sudo netstat -punta | grep 2550 | grep LISTEN    member_index=${index}
+    \    ${listening} =    Get Match    ${output}    LISTEN
+    \    BuiltIn.Run Keyword If    '${listening}' == 'None'    ClusterManagement.Check_Bash_Command_On_Member    command=pid=$(grep org.apache.karaf.main.Main | grep -v grep | tr -s ' ' | cut -f2 -d' '); sudo /usr/lib/jvm/java-1.8.0/bin/jstack -l ${pid}    member_index=${index}
+    BuiltIn.Wait_Until_Keyword_Succeeds    60s    10s    ShardStability.Shards_Stability_Get_Details    ${DEFAULT_SHARD_LIST}    verify_restconf=True
+
+*** Keywords ***
+Get Match
+    [Arguments]    ${text}    ${regexp}    ${index}=0
+    [Documentation]    Wrapper around String.Get Regexp Matches to return None if not found or the first match if found.
+    @{matches} =    String.Get Regexp Matches    ${text}    ${regexp}
+    ${matches_length} =    BuiltIn.Get Length    ${matches}
+    BuiltIn.Set Suite Variable    ${OS_MATCH}    None
+    BuiltIn.Run Keyword If    ${matches_length} > ${index}    BuiltIn.Set Suite Variable    ${OS_MATCH}    @{matches}[${index}]
+    [Return]    ${OS_MATCH}