Verify restconf is available on member startup

CSIT run
https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/controller-csit-3node-clustering-tell-all-neon/106/robot-plugin/log.html.gz
failed due to the restconf availability check returning 404.
This occurred at 17:16:24 however the karaf log shows that
the restconf bundle was still in the process of starting up
at that time (it was waiting for ia AAA dependency). The
bundle eventually completed startup at 17:17:19.

The restconf check is part of the
Get_Raft_Property_From_Shard_Member keyword. However it's
wrapped in the higher-level
Verify_Leader_Exists_For_Each_Shard keyword with a deadline
of 30 sec which is enough most of the time but needs a bigger
cushion. Unfortunately, since the restconf check is deeply
nested within a Wait Until Keyword Succeeds, we can't just
simply wrap it with it's own deadline as that may be longer
than the enclosing deadline. Instead, I added the restconf
check to the Start_Members_From_List_Or_All keyword which
already has a 5 min deadline. I left
Get_Raft_Property_From_Shard_Member as is lthough I really
don't think the restconf check belongs there.

Change-Id: I687a10cbc92d8190a2bcb9a5f4ac1837ba0459a8
Signed-off-by: Tom Pantelis <tompantelis@gmail.com>
Signed-off-by: Jamo Luhrsen <jluhrsen@redhat.com>

diff --git a/csit/libraries/ClusterManagement.robot b/csit/libraries/ClusterManagement.robot
index 69aa47254a952dd2903615115c236ee5e6880a86..f5e4fdf9a3f87525e3b9cf07f4ecc39a507c4886 100644 (file)
--- a/csit/libraries/ClusterManagement.robot
+++ b/csit/libraries/ClusterManagement.robot
@@ -458,7 +458,7 @@ Start_Single_Member
 
 Start_Members_From_List_Or_All
     [Arguments]    ${member_index_list}=${EMPTY}    ${wait_for_sync}=True    ${timeout}=300s    ${karaf_home}=${EMPTY}    ${export_java_home}=${EMPTY}    ${gc_log_dir}=${EMPTY}
-    ...    ${check_system_status}=False    ${service_list}=@{EMPTY}
+    ...    ${check_system_status}=False    ${service_list}=@{EMPTY}    ${verify_restconf}=True
     [Documentation]    If the list is empty, start all cluster members. Otherwise, start members based on present indices.
     ...    If ${wait_for_sync}, wait for cluster sync on listed members.
     ...    Optionally karaf_home can be overriden. Optionally specific JAVA_HOME is used for starting.
@@ -469,12 +469,27 @@ Start_Members_From_List_Or_All
     ${gc_filepath} =    BuiltIn.Set_Variable_If    """${karaf_home}""" != ""    ${karaf_home}/data/log/gc_${epoch}.log    ${GC_LOG_PATH}/gc_${epoch}.log
     ${gc_options} =    BuiltIn.Set_Variable_If    "docker" not in """${node_start_command}"""    -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:${gc_filepath}    ${EMPTY}
     Run_Bash_Command_On_List_Or_All    command=${command} ${gc_options}    member_index_list=${member_index_list}
-    BuiltIn.Return_From_Keyword_If    not ${wait_for_sync}
-    BuiltIn.Wait_Until_Keyword_Succeeds    ${timeout}    10s    Check_Cluster_Is_In_Sync    member_index_list=${member_index_list}
-    BuiltIn.Return_From_Keyword_If    not ${check_system_status}
-    CompareStream.Run_Keyword_If_At_Least_Oxygen    Wait Until Keyword Succeeds    60    2    ClusterManagement.Check Status Of Services Is OPERATIONAL    @{service_list}
+    BuiltIn.Wait_Until_Keyword_Succeeds    ${timeout}    10s    Verify_Members_Are_Ready    ${member_index_list}    ${wait_for_sync}    ${verify_restconf}
+    ...    ${check_system_status}    ${service_list}
     [Teardown]    Run_Bash_Command_On_List_Or_All    command=netstat -pnatu | grep 2550
 
+Verify_Members_Are_Ready
+    [Arguments]    ${member_index_list}    ${verify_cluster_sync}    ${verify_restconf}    ${verify_system_status}    ${service_list}
+    [Documentation]    Verifies the specified readiness conditions for the given listed members after startup.
+    ...    If ${verify_cluster_sync}, verifies the datastores have synced with the rest of the cluster.
+    ...    If ${verify_restconf}, verifies RESTCONF is available.
+    ...    If ${verify_system_status}, verifies the system services are OPERATIONAL.
+    BuiltIn.Run_Keyword_If    ${verify_cluster_sync}    Check_Cluster_Is_In_Sync    ${member_index_list}
+    BuiltIn.Run_Keyword_If    ${verify_restconf}    Verify_Restconf_Is_Available    ${member_index_list}
+    BuiltIn.Run_Keyword_If    ${verify_system_status}    CompareStream.Run_Keyword_If_At_Least_Oxygen    ClusterManagement.Check Status Of Services Is OPERATIONAL    ${service_list}
+
+Verify_Restconf_Is_Available
+    [Arguments]    ${member_index_list}
+    ${index_list} =    List_Indices_Or_All    given_list=${member_index_list}
+    : FOR    ${index}    IN    @{index_list}
+    \    ${session} =    Resolve_Http_Session_For_Member    member_index=${index}
+    \    TemplatedRequests.Get_As_Json_Templated    session=${session}    folder=${RESTCONF_MODULES_DIR}    verify=False
+
 Freeze_Single_Member
     [Arguments]    ${member}
     [Documentation]    Convenience keyword that stops the specified member of the cluster by freezing the jvm.
@@ -874,4 +889,4 @@ Check Status Of Services Is OPERATIONAL
     [Arguments]    @{service_list}
     [Documentation]    This keyword will verify whether all the services are operational in all the ODL nodes
     : FOR    ${i}    IN RANGE    ${NUM_ODL_SYSTEM}
-    \    ClusterManagement.Check Service Status    ${ODL_SYSTEM_${i+1}_IP}    ACTIVE    OPERATIONAL    @{service_list}
+    \    ClusterManagement.Check Service Status    ${ODL_SYSTEM_${i+1}_IP}    ACTIVE    OPERATIONAL    ${service_list}