2 Documentation Switch connections and cluster are restarted.
3 Suite Setup Initialization Phase
4 Suite Teardown Final Phase
5 Test Setup SetupUtils.Setup_Test_With_Logging_And_Without_Fast_Failing
6 Library RequestsLibrary
8 Resource ../../../libraries/SetupUtils.robot
9 Resource ../../../libraries/ClusterManagement.robot
10 Resource ../../../libraries/ClusterOpenFlow.robot
11 Resource ../../../libraries/TemplatedRequests.robot
12 Resource ../../../libraries/MininetKeywords.robot
13 Resource ../../../libraries/Utils.robot
14 Resource ../../../libraries/OvsManager.robot
15 Resource ../../../variables/Variables.robot
16 Resource ../../../variables/openflowplugin/Variables.robot
21 ${VAR_DIR} ${CURDIR}/../../../variables/openflowplugin
25 [Documentation] Add ${ITER} groups 1&2 and flows in every switch.
26 Add Groups And Flows On Member ${ITER}
28 Start Mininet Multiple Connections
29 [Documentation] Start mininet linear with connection to all cluster instances.
30 ${cluster_index_list}= ClusterManagement.List All Indices
31 ${mininet_conn_id}= MininetKeywords.Start Mininet Multiple Controllers ${TOOLS_SYSTEM_IP} ${cluster_index_list} --topo linear,${SWITCHES}
32 BuiltIn.Set Suite Variable ${cluster_index_list}
33 BuiltIn.Set Suite Variable ${mininet_conn_id}
34 BuiltIn.Wait Until Keyword Succeeds 10s 1s OVSDB.Check OVS OpenFlow Connections ${TOOLS_SYSTEM_IP} ${SWITCHES*3}
37 [Documentation] Check Linear Topology.
38 BuiltIn.Wait Until Keyword Succeeds 30s 1s ClusterOpenFlow.Check Linear Topology On Member ${SWITCHES}
40 Check Stats Are Not Frozen
41 [Documentation] Check that duration flow stat is increasing.
42 BuiltIn.Wait Until Keyword Succeeds 30s 1s Check Flow Stats Are Not Frozen
44 Check Flows In Operational DS
45 [Documentation] Check Flows in operational DS.
46 BuiltIn.Wait Until Keyword Succeeds 10s 1s ClusterOpenFlow.Check Number Of Flows On Member ${all_flows}
48 Check Groups In Operational DS
49 [Documentation] Check Groups in operational DS.
50 BuiltIn.Wait Until Keyword Succeeds 10s 1s ClusterOpenFlow.Check Number Of Groups On Member ${all_groups}
53 [Documentation] Check Flows in switch.
54 MininetKeywords.Check Flows In Mininet ${mininet_conn_id} ${all_flows}
56 Check Entity Owner Status And Find Owner and Successor Before Fail
57 [Documentation] Check Entity Owner Status and identify owner and successor for first switch s1.
58 ${original_owner} ${original_successor_list} ClusterOpenFlow.Get OpenFlow Entity Owner Status For One Device openflow:1 1
59 BuiltIn.Set Suite Variable ${original_owner}
60 BuiltIn.Set Suite Variable ${new_cluster_list} ${original_successor_list}
62 Disconnect Mininet From Owner
63 [Documentation] Disconnect mininet from the owner
64 ${original_owner_list} BuiltIn.Create List ${original_owner}
65 MininetKeywords.Disconnect Cluster Mininet break ${original_owner_list}
66 BuiltIn.Set Suite Variable ${original_owner_list}
68 Check Entity Owner Status And Find Owner and Successor After Fail
69 [Documentation] Check Entity Owner Status and identify owner and successor for first switch s1.
70 ${new_owner} ${new_successor_list} BuiltIn.Wait Until Keyword Succeeds 10s 1s ClusterOpenFlow.Get OpenFlow Entity Owner Status For One Device openflow:1
71 ... 1 ${new_cluster_list} after_stop=True
72 ${owner_list}= BuiltIn.Create List ${original_owner} ${new_owner}
73 BuiltIn.Set Suite Variable ${owner_list}
74 BuiltIn.Set Suite Variable ${new_owner}
75 BuiltIn.Set Suite Variable ${new_successor_list}
77 Check Switch Moves To New Master
78 [Documentation] Check switch s1 is connected to new Master.
79 ${new_master}= BuiltIn.Set Variable ${ODL_SYSTEM_${new_owner}_IP}
80 BuiltIn.Wait Until Keyword Succeeds 10s 1s OvsManager.Should Be Master s1 ${new_master} update_data=${True}
82 Check Linear Topology After Disconnect
83 [Documentation] Check Linear Topology.
84 BuiltIn.Wait Until Keyword Succeeds 30s 1s ClusterOpenFlow.Check Linear Topology On Member ${SWITCHES}
86 Check Stats Are Not Frozen After Disconnect
87 [Documentation] Check that duration flow stat is increasing
88 BuiltIn.Wait Until Keyword Succeeds 30s 1s Check Flow Stats Are Not Frozen
90 Remove Flows And Groups After Mininet Is Disconnected
91 [Documentation] Remove 1 group 1&2 and 1 flow in every switch.
92 Remove Single Group And Flow On Member
94 Check Flows In Operational DS After Mininet Is Disconnected
95 [Documentation] Check Flows in Operational DS.
96 BuiltIn.Wait Until Keyword Succeeds 30s 1s ClusterOpenFlow.Check Number Of Flows On Member ${less_flows}
98 Check Groups In Operational DS After Mininet Is Disconnected
99 [Documentation] Check Groups in Operational DS.
100 BuiltIn.Wait Until Keyword Succeeds 10s 1s ClusterOpenFlow.Check Number Of Groups On Member ${less_groups}
102 Check Flows In Switch After Mininet Is Disconnected
103 [Documentation] Check Flows in switch.
104 MininetKeywords.Check Flows In Mininet ${mininet_conn_id} ${less_flows}
106 Reconnect Mininet To Owner
107 [Documentation] Reconnect mininet to switch 1 owner.
108 MininetKeywords.Disconnect Cluster Mininet restore ${original_owner_list}
110 Check Entity Owner Status And Find Owner and Successor After Reconnect
111 [Documentation] Check Entity Owner Status and identify owner and successor for first switch s1.
112 BuiltIn.Wait Until Keyword Succeeds 10s 1s ClusterOpenFlow.Get OpenFlow Entity Owner Status For One Device openflow:1 1
114 Add Flows And Groups After Owner Reconnect
115 [Documentation] Add 1 group type 1&2 and 1 flow in every switch.
116 Add Single Group And Flow On Member
118 Check Stats Are Not Frozen After Owner Reconnect
119 [Documentation] Check that duration flow stat is increasing.
120 BuiltIn.Wait Until Keyword Succeeds 30s 1s Check Flow Stats Are Not Frozen
122 Check Flows After Owner Reconnect In Operational DS
123 [Documentation] Check Flows in Operational DS.
124 BuiltIn.Wait Until Keyword Succeeds 30s 1s ClusterOpenFlow.Check Number Of Flows On Member ${all_flows}
126 Check Groups After Owner Reconnect In Operational DS
127 [Documentation] Check Groups in Operational DS.
128 BuiltIn.Wait Until Keyword Succeeds 10s 1s ClusterOpenFlow.Check Number Of Groups On Member ${all_groups}
130 Check Flows After Owner Reconnect In Switch
131 [Documentation] Check Flows in switch.
132 MininetKeywords.Check Flows In Mininet ${mininet_conn_id} ${all_flows}
134 Check Switches Generate Slave Connection
135 [Documentation] Check switches are connected to new Slave.
136 ${original_master}= BuiltIn.Set Variable ${ODL_SYSTEM_${original_owner}_IP}
137 FOR ${switch} IN RANGE 1 ${switches+1}
138 BuiltIn.Wait Until Keyword Succeeds 50s 1s OvsManager.Should Be Slave s${switch} ${original_master}
139 ... update_data=${True}
142 Disconnect Mininet From Successor
143 [Documentation] Disconnect mininet from the Successor.
144 MininetKeywords.Disconnect Cluster Mininet break ${new_successor_list}
146 Check Entity Owner Status And Find New Owner and Successor After Disconnect
147 [Documentation] Check Entity Owner Status and identify owner and successor for first switch s1.
148 ${current_owner} ${current_successor_list} BuiltIn.Wait Until Keyword Succeeds 10s 1s ClusterOpenFlow.Get OpenFlow Entity Owner Status For One Device openflow:1
149 ... 1 ${owner_list} after_stop=True
150 BuiltIn.Set Suite Variable ${current_owner}
151 BuiltIn.Set Suite Variable ${current_successor_list}
153 Disconnect Mininet From Current Owner
154 [Documentation] Disconnect mininet from the owner
155 ${current_owner_list}= BuiltIn.Create List ${current_owner}
156 MininetKeywords.Disconnect Cluster Mininet break ${current_owner_list}
158 Check Entity Owner Status And Find Current Owner and Successor After Disconnect
159 [Documentation] Check Entity Owner Status and identify owner and successor for first switch s1.
160 ${current_new_owner} ${current_new_successor_list} BuiltIn.Wait Until Keyword Succeeds 10s 1s ClusterOpenFlow.Get OpenFlow Entity Owner Status For One Device openflow:1
161 ... 1 ${original_owner_list} after_stop=True
162 BuiltIn.Set Suite Variable ${current_new_owner}
163 BuiltIn.Set Suite Variable ${current_new_successor_list}
165 Check Switch Moves To Current Master
166 [Documentation] Check switch s1 is connected to original Master.
167 ${current_new_master}= BuiltIn.Set Variable ${ODL_SYSTEM_${current_new_owner}_IP}
168 BuiltIn.Wait Until Keyword Succeeds 10s 1s OvsManager.Should Be Master s1 ${current_new_master} update_data=${True}
169 BuiltIn.Should Be Equal ${current_new_owner} ${original_owner}
171 Check Linear Topology After Owner Disconnect
172 [Documentation] Check Linear Topology.
173 BuiltIn.Wait Until Keyword Succeeds 30s 1s ClusterOpenFlow.Check Linear Topology On Member ${SWITCHES}
175 Check Stats Are Not Frozen After Owner Disconnect
176 [Documentation] Check that duration flow stat is increasing.
177 BuiltIn.Wait Until Keyword Succeeds 30s 1s Check Flow Stats Are Not Frozen
179 Remove Flows And Groups After Owner Disconnected
180 [Documentation] Remove 1 group 1&2 and 1 flow in every switch.
181 Remove Single Group And Flow On Member
183 Check Flows In Operational DS After Owner Disconnected
184 [Documentation] Check Flows in Operational DS.
185 BuiltIn.Wait Until Keyword Succeeds 30s 1s ClusterOpenFlow.Check Number Of Flows On Member ${less_flows}
187 Check Groups In Operational DS After Owner Disconnected
188 [Documentation] Check Groups in Operational DS.
189 BuiltIn.Wait Until Keyword Succeeds 10s 1s ClusterOpenFlow.Check Number Of Groups On Member ${less_groups}
191 Check Flows In Switch After Owner Disconnected
192 [Documentation] Check Flows in switch.
193 MininetKeywords.Check Flows In Mininet ${mininet_conn_id} ${less_flows}
195 Disconnect Mininet From Cluster
196 [Documentation] Disconnect Mininet from Cluster.
197 MininetKeywords.Disconnect Cluster Mininet break ${original_owner_list}
199 Check No Switches After Disconnect
200 [Documentation] Check no switches in topology.
201 BuiltIn.Wait Until Keyword Succeeds 30s 1s ClusterOpenFlow.Check No Switches On Member ${SWITCHES}
203 Check Switch Is Not Connected
204 [Documentation] Check switch s1 is not connected to any controller.
205 FOR ${index} IN @{cluster_index_list}
206 BuiltIn.Wait Until Keyword Succeeds 10s 1s OvsManager.Should Be Disconnected s1 ${ODL_SYSTEM_${index}_IP}
207 ... update_data=${True}
210 Reconnect Mininet To Cluster
211 [Documentation] Reconnect mininet to cluster by removing Iptables drop rules that were used to disconnect
212 MininetKeywords.Disconnect Cluster Mininet restore
214 Check Linear Topology After Mininet Reconnects
215 [Documentation] Check Linear Topology.
216 BuiltIn.Wait Until Keyword Succeeds 10s 1s ClusterOpenFlow.Check Linear Topology On Member ${SWITCHES}
218 Add Flows And Groups After Mininet Reconnects
219 [Documentation] Add 1 group type 1&2 and 1 flow in every switch.
220 Add Single Group And Flow On Member
222 Check Flows In Operational DS After Mininet Reconnects
223 [Documentation] Check Flows in Operational DS.
224 BuiltIn.Wait Until Keyword Succeeds 30s 1s ClusterOpenFlow.Check Number Of Flows On Member ${all_flows}
226 Check Groups In Operational DS After Mininet Reconnects
227 [Documentation] Check Groups in Operational DS.
228 BuiltIn.Wait Until Keyword Succeeds 10s 1s ClusterOpenFlow.Check Number Of Groups On Member ${all_groups}
230 Check Flows In Switch After Mininet Reconnects
231 [Documentation] Check Flows in switch.
232 MininetKeywords.Check Flows In Mininet ${mininet_conn_id} ${all_flows}
234 Check Entity Owner Status And Find Owner and Successor Before Owner Stop
235 [Documentation] Check Entity Owner Status and identify owner and successor for first switch s1.
236 ${original_owner} ${original_successor_list} ClusterOpenFlow.Get OpenFlow Entity Owner Status For One Device openflow:1 1
237 ${original_successor}= Collections.Get From List ${original_successor_list} 0
238 BuiltIn.Set Suite Variable ${original_owner}
239 BuiltIn.Set Suite Variable ${original_successor_list}
240 BuiltIn.Set Suite Variable ${original_successor}
242 Check Switch Generates Slave Connection Before Owner Stop
243 [Documentation] Check switch s1 is connected to Slave.
244 ${original_slave}= BuiltIn.Set Variable ${ODL_SYSTEM_${original_successor}_IP}
245 BuiltIn.Wait Until Keyword Succeeds 50s 1s OvsManager.Should Be Slave s1 ${original_slave} update_data=${True}
247 Check Shards Status Before Owner Stop
248 [Documentation] Check Status for all shards in OpenFlow application.
249 ClusterOpenFlow.Check OpenFlow Shards Status
252 [Documentation] Stop Owner Instance and verify it is shutdown
253 ClusterManagement.Stop Single Member ${original_owner}
254 BuiltIn.Set Suite Variable ${new_cluster_list} ${original_successor_list}
256 Check Shards Status After Stop
257 [Documentation] Check Status for all shards in OpenFlow application.
258 ClusterOpenFlow.Check OpenFlow Shards Status After Cluster Event ${new_cluster_list}
260 Check Entity Owner Status And Find Owner and Successor After Stop
261 [Documentation] Check Entity Owner Status and identify owner and successor.
262 ${new_owner} ${new_successor_list} ClusterOpenFlow.Get OpenFlow Entity Owner Status For One Device openflow:1 ${original_successor} ${new_cluster_list} after_stop=True
263 BuiltIn.Set Suite Variable ${new_owner}
265 Check Stats Are Not Frozen After Owner Stop
266 [Documentation] Check that duration flow stat is increasing in new owner.
267 BuiltIn.Wait Until Keyword Succeeds 30s 1s Check Flow Stats Are Not Frozen ${new_owner}
269 Remove Configuration In Owner and Verify After Owner Stop
270 [Documentation] Remove 1 group 1&2 and 1 flow in every switch.
271 Remove Single Group And Flow On Member ${new_owner}
273 Check Flows After Owner Stop In Operational DS
274 [Documentation] Check Flows in Operational DS in new owner.
275 BuiltIn.Wait Until Keyword Succeeds 30s 1s ClusterOpenFlow.Check Number Of Flows On Member ${less_flows} ${new_owner}
277 Check Groups After Owner Stop In Operational DS
278 [Documentation] Check Groups in Operational DS in new owner.
279 BuiltIn.Wait Until Keyword Succeeds 10s 1s ClusterOpenFlow.Check Number Of Groups On Member ${less_groups} ${new_owner}
281 Check Flows In Switch After Owner Stop
282 [Documentation] Check Flows in switch.
283 MininetKeywords.Check Flows In Mininet ${mininet_conn_id} ${less_flows}
285 Start Old Owner Instance
286 [Documentation] Start old Owner Instance and verify it is up.
287 ClusterManagement.Start Single Member ${original_owner}
289 Check Entity Owner Status And Find Owner and Successor After Start Owner
290 [Documentation] Check Entity Owner Status and identify owner and successor for first switch s1.
291 ${owner} ${successor_list} BuiltIn.Wait Until Keyword Succeeds 10s 1s ClusterOpenFlow.Get OpenFlow Entity Owner Status For One Device openflow:1
294 Check Linear Topology After Owner Restart
295 [Documentation] Check Linear Topology.
296 BuiltIn.Wait Until Keyword Succeeds 10s 1s ClusterOpenFlow.Check Linear Topology On Member ${SWITCHES}
298 Add Configuration In Owner and Verify After Owner Restart
299 [Documentation] Add 1 group type 1&2 and 1 flow in every switch.
300 Add Single Group And Flow On Member ${new_owner}
302 Check Stats Are Not Frozen After Owner Restart
303 [Documentation] Check that duration flow stat is increasing.
305 BuiltIn.Wait Until Keyword Succeeds 30s 1s Check Flow Stats Are Not Frozen
307 Check Flows In Operational DS After Owner Restart
308 [Documentation] Check Flows in Operational DS.
309 BuiltIn.Wait Until Keyword Succeeds 30s 1s ClusterOpenFlow.Check Number Of Flows On Member ${all_flows}
311 Check Groups In Operational DS After Owner Restart
312 [Documentation] Check Groups in Operational DS.
313 BuiltIn.Wait Until Keyword Succeeds 10s 1s ClusterOpenFlow.Check Number Of Groups On Member ${all_groups}
315 Check Flows In Switch After Owner Restart
316 [Documentation] Check Flows in switch.
317 MininetKeywords.Check Flows In Mininet ${mininet_conn_id} ${all_flows}
320 [Documentation] Stop and Start cluster.
321 # Try to stop contoller, if stop does not work or takes too long, kill controller.
322 ${status} ${result}= BuiltIn.Run Keyword And Ignore Error ClusterManagement.Stop_Members_From_List_Or_All
323 BuiltIn.Run Keyword If '${status}' != 'PASS' ClusterManagement.Kill_Members_From_List_Or_All
324 ClusterManagement.Start_Members_From_List_Or_All wait_for_sync=False
326 Check Linear Topology After Controller Restarts
327 [Documentation] Check Linear Topology
328 BuiltIn.Wait Until Keyword Succeeds 300s 2s ClusterOpenFlow.Check Linear Topology On Member ${SWITCHES}
330 Check Stats Are Not Frozen After Cluster Restart
331 [Documentation] Check that duration flow stat is increasing.
332 BuiltIn.Wait Until Keyword Succeeds 30s 1s Check Flow Stats Are Not Frozen
334 Check Flows In Operational DS After Controller Restarts
335 [Documentation] Check Flows in Operational DS.
336 BuiltIn.Wait Until Keyword Succeeds 30s 1s ClusterOpenFlow.Check Number Of Flows On Member ${all_flows}
338 Check Groups In Operational DS After Controller Restarts
339 [Documentation] Check Groups in Operational DS.
340 BuiltIn.Wait Until Keyword Succeeds 10s 1s ClusterOpenFlow.Check Number Of Groups On Member ${all_groups}
342 Check Flows In Switch After Controller Restarts
343 [Documentation] Check Flows in switch.
344 MininetKeywords.Check Flows In Mininet ${mininet_conn_id} ${all_flows}
347 [Documentation] Stop Mininet.
348 MininetKeywords.Stop Mininet And Exit
351 [Documentation] Check no switches in topology.
352 BuiltIn.Wait Until Keyword Succeeds 5s 1s ClusterOpenFlow.Check No Switches On Member ${SWITCHES}
356 [Documentation] Create controller session and set variables.
357 SetupUtils.Setup_Utils_For_Setup_And_Teardown
358 RequestsLibrary.Create Session session http://${ODL_SYSTEM_IP}:${RESTCONFPORT} auth=${AUTH}
359 ${switches} Convert To Integer ${SWITCHES}
360 ${iter} Convert To Integer ${ITER}
361 ${all_groups}= BuiltIn.Evaluate ${switches} * ${iter} * 2
362 ${less_groups}= BuiltIn.Evaluate ${all_groups} - ${switches} * 2
363 # Stale flows/groups feature enabled in Boron onwards.
364 ${all_flows}= BuiltIn.Evaluate ${switches} * ${iter+1}
365 ${less_flows}= BuiltIn.Evaluate ${all_flows} - ${switches}
366 # Stale flows/groups feature enabled in Boron onwards.
367 BuiltIn.Set Suite Variable ${switches}
368 BuiltIn.Set Suite Variable ${iter}
369 BuiltIn.Set Suite Variable ${all_groups}
370 BuiltIn.Set Suite Variable ${less_groups}
371 BuiltIn.Set Suite Variable ${all_flows}
372 BuiltIn.Set Suite Variable ${less_flows}
373 BuiltIn.Set Suite Variable ${no_flows} ${SWITCHES}
376 [Documentation] Delete all sessions.
377 ${command} = BuiltIn.Set Variable sudo iptables -v -F
378 BuiltIn.Run Keyword And Ignore Error ClusterManagement.Run_Bash_Command_On_List_Or_All ${command}
379 BuiltIn.Run Keyword And Ignore Error RequestsLibrary.Delete Request session ${RFC8040_NODES_API}
380 RequestsLibrary.Delete All Sessions
382 Add Groups And Flows On Member
383 [Arguments] ${iter}=1 ${member_index}=1
384 [Documentation] Add ${ITER} groups type 1 & 2 and flows in every switch.
385 ${session} = Resolve_Http_Session_For_Member member_index=${member_index}
386 FOR ${switch} IN RANGE 1 ${switches+1}
387 &{mapping} BuiltIn.Create_Dictionary NODE=openflow:${switch}
388 TemplatedRequests.Post As Json Templated folder=${VAR_DIR}/add-group-1 mapping=${mapping} session=${session} iterations=${iter}
389 TemplatedRequests.Post As Json Templated folder=${VAR_DIR}/add-group-2 mapping=${mapping} session=${session} iterations=${iter}
390 TemplatedRequests.Post As Json Templated folder=${VAR_DIR}/add-flow mapping=${mapping} session=${session} iterations=${iter}
393 Add Single Group And Flow On Member
394 [Arguments] ${member_index}=1
395 [Documentation] Add 1 group 1&2 and 1 flow in every switch.
396 Add Groups And Flows On Member 1 ${member_index}
398 Remove Single Group And Flow On Member
399 [Arguments] ${member_index}=1
400 [Documentation] Remove 1 group 1&2 and 1 flow in every switch.
401 ${session} = Resolve_Http_Session_For_Member member_index=${member_index}
402 FOR ${switch} IN RANGE 1 ${switches+1}
403 RequestsLibrary.Delete Request ${session} ${RFC8040_NODES_API}/node=openflow%3A${switch}/flow-node-inventory:table=0/flow=1
404 RequestsLibrary.Delete Request ${session} ${RFC8040_NODES_API}/node=openflow%3A${switch}/flow-node-inventory:group=1
405 RequestsLibrary.Delete Request ${session} ${RFC8040_NODES_API}/node=openflow%3A${switch}/flow-node-inventory:group=1000
408 Check Flow Stats Are Not Frozen
409 [Arguments] ${member_index}=1 ${period_in_seconds}=5
410 [Documentation] Verify flow stats are not frozen for flow 1 and switch 1.
411 ${duration_1} = Extract Flow Duration ${member_index}
412 ${duration_1} Builtin.Convert To Integer ${duration_1}
413 BuiltIn.Sleep ${period_in_seconds}
414 ${duration_2} = Extract Flow Duration ${member_index}
415 ${duration_2} Builtin.Convert To Integer ${duration_2}
416 Should Not Be Equal As Integers ${duration_1} ${duration_2}
418 Extract Flow Duration
419 [Arguments] ${member_index}
420 [Documentation] Extract duration for flow 1 in switch 1.
421 ${session} = Resolve_Http_Session_For_Member member_index=${member_index}
422 ${resp} RequestsLibrary.Get Request ${session} ${RFC8040_NODES_API}/node=openflow%3A1/flow-node-inventory:table=0/flow=1?content=nonconfig headers=${headers}
424 ${json_resp} = RequestsLibrary.To_Json ${resp.content}
425 ${flow_list} = Collections.Get_From_Dictionary ${json_resp} flow-node-inventory:flow
426 ${flow_stats} = Collections.Get_From_Dictionary ${flow_list}[0] opendaylight-flow-statistics:flow-statistics
427 ${duration} = Collections.Get_From_Dictionary ${flow_stats}[duration] second
428 Return From Keyword ${duration}