+ ${index_list} = ClusterManagement.List_Indices_Or_All given_list=${member_index_list}
+ : FOR ${index} IN @{index_list} # usually: 1, 2, 3.
+ \ Execute_Controller_Karaf_Command_On_Background log:set ${bgpcep_level} org.opendaylight.bgpcep member_index=${index}
+ \ Execute_Controller_Karaf_Command_On_Background log:set ${protocol_level} org.opendaylight.protocol member_index=${index}
+
+Get Karaf Log Lines From Test Start
+ [Arguments] ${ip} ${test_name} ${cmd} ${user}=${ODL_SYSTEM_USER} ${password}=${ODL_SYSTEM_PASSWORD} ${prompt}=${ODL_SYSTEM_PROMPT}
+ ... ${log_file}=${KARAF_LOG}
+ [Documentation] Scrapes all log messages that match regexp ${type} which fall after a point given by a log message that
+ ... contains ${test_name}. This is useful if your test cases are marking karaf.log with a message indicating when
+ ... that test case has started; such that you can easily pull out any extra log messsages to parse/log/etc in the
+ ... test logic itself. For example, you can grab all ERRORS that occur during your test case.
+ ${output} = Run Command On Controller ${ip} ${cmd} ${user} ${password} ${prompt}
+ @{log_lines} = Split String ${output} ${\n}
+ [Return] ${log_lines}
+
+Fail If Exceptions Found During Test
+ [Arguments] ${test_name} ${log_file}=${KARAF_LOG} ${fail}=False
+ [Documentation] Create a failure if an Exception is found in the karaf.log that has not been whitelisted.
+ ... Will work for single controller jobs as well as 3node cluster jobs
+ : FOR ${i} IN RANGE 1 ${NUM_ODL_SYSTEM} + 1
+ \ ${cmd} = Set Variable sed '1,/ROBOT MESSAGE: Starting test ${test_name}/d' ${log_file}
+ \ ${output} = Get Karaf Log Lines From Test Start ${ODL_SYSTEM_${i}_IP} ${test_name} ${cmd}
+ \ ${exlist} ${matchlist} = Verify Exceptions ${output}
+ \ Write Exceptions Map To File ${SUITE_NAME}.${TEST_NAME} /tmp/odl${i}_exceptions.txt
+ \ ${listlength} = BuiltIn.Get Length ${exlist}
+ \ BuiltIn.Run Keyword If "${fail}"=="True" and ${listlength} != 0 Log And Fail Exceptions ${exlist} ${listlength}
+ \ ... ELSE Collections.Log List ${matchlist}
+
+Log And Fail Exceptions
+ [Arguments] ${exlist} ${listlength}
+ [Documentation] Print the list of failed exceptions and fail the test
+ Collections.Log List ${exlist}
+ ${exstr} = BuiltIn.Catenate ${exlist}
+ BuiltIn.Fail New exceptions found: ${listlength}\n${exstr}
+
+Get Karaf Log Type From Test Start
+ [Arguments] ${ip} ${test_name} ${type} ${user}=${ODL_SYSTEM_USER} ${password}=${ODL_SYSTEM_PASSWORD} ${prompt}=${ODL_SYSTEM_PROMPT}
+ ... ${log_file}=${KARAF_LOG}
+ [Documentation] Scrapes all log messages that match regexp ${type} which fall after a point given by a log message that
+ ... contains ${test_name}. This is useful if your test cases are marking karaf.log with a message indicating when
+ ... that test case has started; such that you can easily pull out any extra log messsages to parse/log/etc in the
+ ... test logic itself. For example, you can grab all ERRORS that occur during your test case.
+ ${cmd} Set Variable sed '1,/ROBOT MESSAGE: Starting test ${test_name}/d' ${log_file} | grep '${type}'
+ ${output} Run Command On Controller ${ip} ${cmd} ${user} ${password} ${prompt}
+ [Return] ${output}
+
+Get Karaf Log Types From Test Start
+ [Arguments] ${ip} ${test_name} ${types} ${user}=${ODL_SYSTEM_USER} ${password}=${ODL_SYSTEM_PASSWORD} ${prompt}=${ODL_SYSTEM_PROMPT}
+ ... ${log_file}=${KARAF_LOG}
+ [Documentation] A wrapper keyword for "Get Karaf Log Type From Test Start" so that we can parse for multiple types
+ ... of log messages. For example, we can grab all messages of type WARN and ERROR
+ : FOR ${type} IN @{types}
+ \ Get Karaf Log Type From Test Start ${ip} ${test_name} ${type} ${user} ${password}
+ \ ... ${prompt} ${log_file}
+
+Get Karaf Log Events From Test Start
+ [Arguments] ${test_name} ${user}=${ODL_SYSTEM_USER} ${password}=${ODL_SYSTEM_PASSWORD} ${prompt}=${ODL_SYSTEM_PROMPT}
+ [Documentation] Wrapper for the wrapper "Get Karaf Log Types From Test Start" so that we can easily loop over
+ ... any number of controllers to analyze karaf.log for ERROR, WARN and Exception log messages
+ ${log_types} = Create List ERROR WARN Exception
+ : FOR ${i} IN RANGE 1 ${NUM_ODL_SYSTEM} + 1
+ \ Get Karaf Log Types From Test Start ${ODL_SYSTEM_${i}_IP} ${test_name} ${log_types}
+
+Fail If Exceptions Found During Test Deprecated
+ [Arguments] ${test_name} ${exceptions_white_list}=${EMPTY}
+ [Documentation] Create a failure if an Exception is found in the karaf.log. Will work for single controller jobs
+ ... as well as 3node cluster jobs
+ : FOR ${i} IN RANGE 1 ${NUM_ODL_SYSTEM} + 1
+ \ Verify Exception Logging In Controller ${ODL_SYSTEM_${i}_IP} ${test_name} ${exceptions_white_list}
+
+Verify Exception Logging In Controller
+ [Arguments] ${controller_ip} ${test_name} ${exceptions_white_list}
+ [Documentation] Local keyword to make it easier to loop through N controllers to pull Exceptions from the
+ ... karaf.log file and validate with "Check Against White List"
+ ${exceptions}= Get Karaf Log Type From Test Start ${controller_ip} ${test_name} Exception
+ @{log_lines}= Split String ${exceptions} ${\n}
+ ${num_log_entries} Get Length ${log_lines}
+ Return From Keyword If ${num_log_entries} == ${0} No Exceptions found.
+ : FOR ${log_message} IN @{log_lines}
+ \ Check Against White List ${log_message} ${exceptions_white_list}
+
+Check Against White List
+ [Arguments] ${exception_line} ${exceptions_white_list}
+ [Documentation] As soon as the ${exceptions_line} is found in one of the elements of ${exceptions_white_list}
+ ... this keyword will exit and give a Pass to the caller. If there is no match, this keyword will end up
+ ... marking a failure. In the case that no exceptions are found, the caller could end up passing a single
+ ... empty line as that is what is returned when a grep on karaf.log has no match, so we can safely return
+ ... in that case as well.
+ Return From Keyword If "${exception_line}" == ""
+ : FOR ${exception} IN @{exceptions_white_list}
+ \ Return From Keyword If "${exception}" in "${exception_line}" Exceptions found, but whitelisted: ${\n}${exception_line}${\n}
+ Fail Exceptions Found: ${\n}${exception_line}${\n}
+
+Wait_For_Karaf_Log
+ [Arguments] ${message} ${timeout}=60 ${member_index}=${1}
+ [Documentation] Read karaf logs until message appear
+ # TODO: refactor this keyword to use the new workflow to account for multiple controllers. Initial work was done
+ # in this patch https://git.opendaylight.org/gerrit/#/c/45596/
+ # however, the consumers of this keyword were breaking after that change. Initial theory is that a previous
+ # keyword used before this "Wait For Karaf Log" keyword was closing the karaf console connection, so the
+ # "Flexible SSH Login" keyword from the patch above (45596) was failing.
+ BuiltIn.Log Waiting for '${message}' in karaf log
+ SSHLibrary.Open_Connection ${ODL_SYSTEM_IP} port=${KARAF_SHELL_PORT} prompt=${KARAF_PROMPT_LOGIN} timeout=${timeout}
+ SSHLibrary.Login ${KARAF_USER} ${KARAF_PASSWORD} loglevel=${loglevel}
+ SSHLibrary.Write log:tail
+ SSHLibrary.Read_Until ${message}
+ SSHLibrary.Write logout
+ SSHLibrary.Close_Connection
+
+Restart_Bundle
+ [Arguments] ${bundle_id}
+ [Documentation] Restarts bundle passed as argument. Note this operation is only for testing and not production environments
+ # TODO: prepare this for cluster environment and multiple controllers
+ Execute_Controller_Karaf_Command_With_Retry_On_Background bundle:restart -f $(bundle:id '${bundle_id}')
+
+Restart_Karaf
+ [Documentation] Restarts Karaf and polls log to detect when Karaf is up and running again
+ # TODO: prepare this for cluster environment and multiple controllers
+ Execute_Controller_Karaf_Command_With_Retry_On_Background log:clear
+ Execute_Controller_Karaf_Command_With_Retry_On_Background shutdown -r -f
+ BuiltIn.Run_Keyword_And_Return_Status BuiltIn.Wait_Until_Keyword_Succeeds 240s 60s Wait_For_Karaf_Log Karaf started in
+
+Restart_Jetty
+ [Documentation] Restarts jetty bundle (to reload certificates or key/truststore information)
+ Execute_Controller_Karaf_Command_With_Retry_On_Background log:clear
+ Restart_Bundle OPS4J Pax Web - Jetty
+ Wait_For_Karaf_Log Instantiated the Application class org.opendaylight.restconf.RestconfApplication
+ Wait_For_Karaf_Log Instantiated the Application class org.opendaylight.netconf.sal.rest.impl.RestconfApplication
+ Wait_For_Karaf_Log Instantiated the Application class org.opendaylight.aaa.idm.IdmLightApplication