Debug help for triaging missing links in CI

The lithium redesign tests are failing because it's not
seeing that a link has been removed (mininet: link s1 s2 down).
I cannot reproduce this locally, but it's happening every time
in CI.  I'm using same OVS/Mininet versions.  At a loss now, so
pushing some steps to maybe help debug.

I've also reduced a few WUKS in other suites from 30s to
6s which is more reasonable and allows us to fail sooner
as 3s should be good enough.  after 6s we can fail.

Change-Id: Ib1178efb2f49b1c8bd1619c60ea3df543203fe9c
Signed-off-by: Jamo Luhrsen <james.luhrsen@hp.com>

diff --git a/test/csit/suites/openflowplugin/MD_SAL_NSF_OF10/020__restconf_topology.robot b/test/csit/suites/openflowplugin/MD_SAL_NSF_OF10/020__restconf_topology.robot
index 13a214eeaff4968dfd4bf2c5102d2d559bc445e0..fc67afdddab5cf8fc288ea963c647d29ca5203f9 100644 (file)
--- a/test/csit/suites/openflowplugin/MD_SAL_NSF_OF10/020__restconf_topology.robot
+++ b/test/csit/suites/openflowplugin/MD_SAL_NSF_OF10/020__restconf_topology.robot
@@ -45,7 +45,17 @@ Link Down
     Write    link s1 s2 down
     Read Until    mininet>
     ${links}    Create List    ${link2}    ${link4}
-    Wait Until Keyword Succeeds    30s    2s    Verify Links    ${links}
+    # increasing the WUKS timeout to 60s to see if the CI environment might just be taking
+    # longer for this test with the lithium redesign plugin
+    Wait Until Keyword Succeeds    60s    2s    Verify Links    ${links}
+    # shot in the dark.  maybe the "link s1 s2 down" really didn't take the link(s) down?
+    # hopefully this output below will show that.
+    Write    sh ovs-vsctl find Interface name="s1-eth1"
+    ${output}=    Read Until    mininet>
+    Log    ${output}
+    Write    sh ovs-vsctl find Interface name="s1-eth2"
+    ${output}=    Read Until    mininet>
+    Log    ${output}
 
 Link Up
     [Documentation]    Take link s1-s2 up

diff --git a/test/csit/suites/openflowplugin/MD_SAL_NSF_OF10/040__restconf_frm.robot b/test/csit/suites/openflowplugin/MD_SAL_NSF_OF10/040__restconf_frm.robot
index 6eb00856cc3344a910460da2146f4f492d29297e..850402e96c9742bc1441e1850c26939e2d9cf316 100644 (file)
--- a/test/csit/suites/openflowplugin/MD_SAL_NSF_OF10/040__restconf_frm.robot
+++ b/test/csit/suites/openflowplugin/MD_SAL_NSF_OF10/040__restconf_frm.robot
@@ -27,7 +27,7 @@ Verify after adding flow config - Output to physical port#
 Verify after adding flow operational - Output to physical port#
     [Documentation]    Verify the flow
     ${elements}=    Create List    10.0.20.1
-    Wait Until Keyword Succeeds    60s    2s    Check For Elements At URI    ${OPERATIONAL_NODES_API}/node/openflow:1/table/0/flow/152    ${elements}
+    Wait Until Keyword Succeeds    6s    2s    Check For Elements At URI    ${OPERATIONAL_NODES_API}/node/openflow:1/table/0/flow/152    ${elements}
 
 Remove a flow - Output to physical port#
     [Documentation]    Remove a flow
@@ -44,4 +44,4 @@ Verify after deleting flow config - Output to physical port#
 Verify after deleting flow operational - Output to physical port#
     [Documentation]    Verify the flow
     ${elements}=    Create List    10.0.20.1
-    Wait Until Keyword Succeeds    60s    2s    Check For Elements Not At URI    ${OPERATIONAL_NODES_API}/node/openflow:1/table/0    ${elements}
+    Wait Until Keyword Succeeds    6s    2s    Check For Elements Not At URI    ${OPERATIONAL_NODES_API}/node/openflow:1/table/0    ${elements}

diff --git a/test/csit/suites/openflowplugin/MD_SAL_NSF_OF13/020__restconf_topology.robot b/test/csit/suites/openflowplugin/MD_SAL_NSF_OF13/020__restconf_topology.robot
index 13a214eeaff4968dfd4bf2c5102d2d559bc445e0..fc67afdddab5cf8fc288ea963c647d29ca5203f9 100644 (file)
--- a/test/csit/suites/openflowplugin/MD_SAL_NSF_OF13/020__restconf_topology.robot
+++ b/test/csit/suites/openflowplugin/MD_SAL_NSF_OF13/020__restconf_topology.robot
@@ -45,7 +45,17 @@ Link Down
     Write    link s1 s2 down
     Read Until    mininet>
     ${links}    Create List    ${link2}    ${link4}
-    Wait Until Keyword Succeeds    30s    2s    Verify Links    ${links}
+    # increasing the WUKS timeout to 60s to see if the CI environment might just be taking
+    # longer for this test with the lithium redesign plugin
+    Wait Until Keyword Succeeds    60s    2s    Verify Links    ${links}
+    # shot in the dark.  maybe the "link s1 s2 down" really didn't take the link(s) down?
+    # hopefully this output below will show that.
+    Write    sh ovs-vsctl find Interface name="s1-eth1"
+    ${output}=    Read Until    mininet>
+    Log    ${output}
+    Write    sh ovs-vsctl find Interface name="s1-eth2"
+    ${output}=    Read Until    mininet>
+    Log    ${output}
 
 Link Up
     [Documentation]    Take link s1-s2 up

diff --git a/test/csit/suites/openflowplugin/MD_SAL_NSF_OF13/030__restconf_statistics.robot b/test/csit/suites/openflowplugin/MD_SAL_NSF_OF13/030__restconf_statistics.robot
index d225cea97593b60fa7b93ec7bbc244e6c825aad9..5a8d0e297b093dcd11afeddd053beaf783063588 100644 (file)
--- a/test/csit/suites/openflowplugin/MD_SAL_NSF_OF13/030__restconf_statistics.robot
+++ b/test/csit/suites/openflowplugin/MD_SAL_NSF_OF13/030__restconf_statistics.robot
@@ -16,6 +16,7 @@ ${REST_CONTEXT}    /restconf/operational/opendaylight-inventory:nodes
 Get Stats for all nodes
     [Documentation]    Get the stats for all nodes
     Wait Until Keyword Succeeds    30s    2s    Ensure All Nodes Are In Response    ${REST_CONTEXT}    ${node_list}
+
 Get Stats for node 1
     [Documentation]    Get the stats for a node
     Wait Until Keyword Succeeds    60s    2s    Check Nodes Stats    openflow:1

diff --git a/test/csit/suites/openflowplugin/MD_SAL_NSF_OF13/040__restconf_frm.robot b/test/csit/suites/openflowplugin/MD_SAL_NSF_OF13/040__restconf_frm.robot
index 197e26c93c2dddf7be1360d0ed11a33c0c0713ca..8f55e88222121ed742b348bedfa7e2c2b1605159 100644 (file)
--- a/test/csit/suites/openflowplugin/MD_SAL_NSF_OF13/040__restconf_frm.robot
+++ b/test/csit/suites/openflowplugin/MD_SAL_NSF_OF13/040__restconf_frm.robot
@@ -28,7 +28,7 @@ Verify after adding flow config - Sending IPv4 Dest Address and Eth type
 Verify after adding flow operational - Sending IPv4 Dest Address and Eth type
     [Documentation]    Verify the flow
     ${elements}=    Create List    10.0.10.1
-    Wait Until Keyword Succeeds    60s    2s    Check For Elements At URI    ${OPERATIONAL_NODES_API}/node/openflow:1/table/2/flow/139    ${elements}
+    Wait Until Keyword Succeeds    6s    2s    Check For Elements At URI    ${OPERATIONAL_NODES_API}/node/openflow:1/table/2/flow/139    ${elements}
 
 Modify a flow - Output to physical port#
     [Documentation]    Push a flow through REST-API
@@ -44,7 +44,7 @@ Verify after modifying flow config - Output to physical port#
 Verify after modifying flow operational - Output to physical port#
     [Documentation]    Verify the flow
     ${elements}=    Create List    10.0.20.1
-    Wait Until Keyword Succeeds    90s    2s    Check For Elements At URI    ${OPERATIONAL_NODES_API}/node/openflow:1/table/2/flow/139    ${elements}
+    Wait Until Keyword Succeeds    6s    2s    Check For Elements At URI    ${OPERATIONAL_NODES_API}/node/openflow:1/table/2/flow/139    ${elements}
 
 Remove a flow - Output to physical port#
     [Documentation]    Remove a flow
@@ -60,4 +60,4 @@ Verify after deleting flow config - Output to physical port#
 Verify after deleting flow operational - Output to physical port#
     [Documentation]    Verify the flow
     ${elements}=    Create List    10.0.20.1
-    Wait Until Keyword Succeeds    60s    2s    Check For Elements Not At URI    ${OPERATIONAL_NODES_API}/node/openflow:1/table/2    ${elements}
+    Wait Until Keyword Succeeds    6s    2s    Check For Elements Not At URI    ${OPERATIONAL_NODES_API}/node/openflow:1/table/2    ${elements}