5 Resource DataModels.robot
8 Resource SSHKeywords.robot
10 Resource ../variables/netvirt/Variables.robot
11 Resource ../variables/Variables.robot
12 Resource VpnOperations.robot
13 Variables ../variables/coe/Modules.py
14 Variables ../variables/netvirt/Modules.py
15 Resource ToolsSystem.robot
18 ${BUSY_BOX} ${CURDIR}/../variables/coe/busy-box.yaml
19 ${CNI_BINARY_FILE} /opt/cni/bin/odlovs-cni
20 ${CONFIG_FILE} /etc/cni/net.d/odlovs-cni.conf
21 ${CONFIG_FILE_TEMPLATE} ${CURDIR}/../variables/coe/odlovs-cni.conf.j2
22 ${HOST_INVENTORY} ${CURDIR}/../variables/coe/hosts.yaml
23 ${K8s_MASTER_IP} ${TOOLS_SYSTEM_1_IP}
24 ${K8s_MINION1_IP} ${TOOLS_SYSTEM_2_IP}
25 ${K8s_MINION2_IP} ${TOOLS_SYSTEM_3_IP}
26 ${K8s_MINION3_IP} ${TOOLS_SYSTEM_4_IP}
27 ${K8s_MINION4_IP} ${TOOLS_SYSTEM_5_IP}
28 ${HOSTS_FILE_TEMPLATE} ${CURDIR}/../variables/coe/minions_template.yaml
29 ${NODE_READY_STATUS} \\sReady
30 ${PLAYBOOK_FILE} ${CURDIR}/../variables/coe/coe_play.yaml
31 ${POD_RUNNING_STATUS} \\sRunning
32 ${VARIABLES_PATH} ${CURDIR}/../variables/coe
33 ${WATCHER_COE} ${CURDIR}/../variables/coe/coe.yaml
34 @{NODE_IPs} ${K8s_MASTER_IP} ${K8s_MINION1_IP} ${K8s_MINION2_IP} ${K8s_MINION3_IP} ${K8s_MINION4_IP}
35 @{COE_DIAG_SERVICES} OPENFLOW IFM ITM DATASTORE ELAN OVSDB
39 [Documentation] COE project requires start suite to be executed only for the first test suite.This keyword find the current suite,compares it with the stored first suite value and executes Coe.Start suite only if the cuurent suite is equal to the first suite.
40 ToolsSystem.Get Tools System Nodes Data
41 Coe.Set Connection ids and Bridge
42 ${current suite} ${suite names updated} Extract current suite name
43 ${first_suite} = Set Variable ${suite names updated[0]}
44 ${status} = BuiltIn.Evaluate '${first_suite}' == '${current suite}'
45 Run Keyword If '${status}' == 'True' Coe.Start Suite
48 [Documentation] Suite setup keyword.
49 Coe.Configuration Playbook
50 Coe.Verify Config Files
51 Coe.Verify Watcher Is Running
52 BuiltIn.Wait Until Keyword Succeeds 40s 2s Coe.Check Node Status Is Ready
54 BuiltIn.Wait Until Keyword Succeeds 60 2 ClusterManagement.Check Status Of Services Is OPERATIONAL @{COE_DIAG_SERVICES}
55 BuiltIn.Wait Until Keyword Succeeds 85 2 Genius.Verify Tunnel Status as UP default-transport-zone
56 Coe.Derive Coe Data Models
58 Set Connection ids and Bridge
59 [Documentation] Sets the connection ids for all the nodes and get the bridge from configuration file .
60 : FOR ${conn_id} IN @{TOOLS_SYSTEM_ALL_CONN_IDS}
61 \ SSHLibrary.Switch Connection ${conn_id}
62 \ SSHKeywords.Flexible_SSH_Login ${DEFAULT_USER} ${DEFAULT_PASSWORD}
63 ${file} = OperatingSystem.Get File ${CONFIG_FILE_TEMPLATE}
64 ${ovs bridge output} ${bridge} = Should Match Regexp ${file} "ovsBridge": "(\\w.*)"
65 Set Suite Variable ${bridge}
67 Configuration Playbook
68 [Documentation] Ansible playbook which does all basic configuration for kubernetes nodes.
69 ${playbook minions} ${playbook hosts} ${host file} Modifying templates in playbook
70 ${playbook} = OperatingSystem.Get File ${PLAYBOOK_FILE}
71 ${playbook} = String.Replace String ${playbook} coe-hosts ${playbook hosts}
72 ${playbook} = String.Replace String ${playbook} coe-minions ${playbook minions}
73 OperatingSystem.Create File ${PLAYBOOK_FILE} ${playbook}
74 OperatingSystem.Create File ${USER_HOME}/hosts.yaml ${host file}
75 ${watcher} = OperatingSystem.Get File ${WATCHER_COE}
76 ${watcher} = String.Replace String ${watcher} odlip ${ODL_SYSTEM_IP}
77 ${watcher} = String.Replace String ${watcher} port ${RESTCONFPORT}
78 OperatingSystem.Create File ${WATCHER_COE} ${watcher}
79 SSHKeywords.Copy_File_To_Remote_System ${K8s_MASTER_IP} ${WATCHER_COE} ${USER_HOME}
80 OperatingSystem.Copy File ${PLAYBOOK_FILE} ${USER_HOME}
81 ${play_output} = OperatingSystem.Run ansible-playbook ${USER_HOME}/coe_play.yaml -i ${USER_HOME}/hosts.yaml
82 BuiltIn.Log ${play_output}
84 Modifying templates in playbook
85 ${inventory} = OperatingSystem.Get File ${HOST_INVENTORY}
86 ${template} = OperatingSystem.Get File ${HOSTS_FILE_TEMPLATE}
87 ${template} = String.Replace String ${template} minion_ip ${TOOLS_SYSTEM_ALL_IPS[0]}
88 @{minions} Create List coe-minion
89 ${hosts} Set Variable coe-master:
90 : FOR ${i} INRANGE 1 ${NUM_TOOLS_SYSTEM}
91 \ Append To List ${minions} coe-minion${i}
92 \ ${hosts} = Catenate ${hosts} coe-minion${i}:
93 ${hosts} = Replace String Using Regexp ${hosts} :$ ${EMPTY}
94 ${hosts} = Remove Space on String ${hosts}
95 ${minion hosts} = Replace String Using Regexp ${hosts} ^[\\w-]+: ${EMPTY}
96 : FOR ${i} INRANGE 1 ${NUM_TOOLS_SYSTEM}
97 \ ${j} = Evaluate ${i}+1
98 \ ${template} = String.Replace String ${template} ${minions[${i}-1]} ${minions[${i}]}
99 \ ${template} = String.Replace String ${template} ${TOOLS_SYSTEM_ALL_IPS[${i}-1]} ${TOOLS_SYSTEM_ALL_IPS[${i}]}
100 \ ${template} = String.Replace String ${template} 192.168.50.1${i} 192.168.50.1${j}
101 \ ${template} = String.Replace String ${template} 10.11.${i}.0/24 10.11.${j}.0/24
102 \ ${template} = String.Replace String ${template} 10.11.${i}.1 10.11.${j}.1
103 \ Append To File ${HOST_INVENTORY} ${template}
104 ${host file} = OperatingSystem.Get File ${HOST_INVENTORY}
105 ${host file} = String.Replace String ${host file} master_ip ${TOOLS_SYSTEM_ALL_IPS[0]}
106 ${host file} = String.Replace String ${host file} odl_ip ${ODL_SYSTEM_IP}
107 ${host file} = String.Replace String ${host file} mport ${OVSDBPORT}
108 ${host file} = String.Replace String ${host file} cport ${ODL_OF_PORT_6653}
109 ${host file} = String.Replace String ${host file} filepath ${CONFIG_FILE_TEMPLATE}
110 ${host file} = String.Replace String ${host file} yamlpath ${USER_HOME}/coe.yaml
112 [Return] ${minion hosts} ${hosts} ${host file}
115 [Documentation] Checks if the configuration files are present in all nodes
116 : FOR ${nodes} IN @{TOOLS_SYSTEM_ALL_IPS}
117 \ Utils.Verify File Exists On Remote System ${nodes} ${CONFIG_FILE}
118 : FOR ${nodes} IN @{TOOLS_SYSTEM_ALL_IPS}
119 \ Utils.Verify File Exists On Remote System ${nodes} ${CNI_BINARY_FILE}
121 Verify Watcher Is Running
122 [Documentation] Checks if watcher is running in the background
123 ${watcher status} = Utils.Run Command On Remote System ${K8s_MASTER_IP} ps -ef | grep watcher
124 BuiltIn.Should Match Regexp ${watcher status} .* watcher odl
126 Check Node Status Is Ready
127 [Documentation] Checks the status of nodes.This keyword is repeated until the status of all nodes is Ready
128 ${nodes} = Utils.Run Command On Remote System ${K8s_MASTER_IP} kubectl get nodes ${DEFAULT_USER} ${DEFAULT_PASSWORD} ${DEFAULT_LINUX_PROMPT_STRICT}
129 @{cluster} = String.Split To Lines ${nodes} 1
130 : FOR ${node} IN @{cluster}
131 \ BuiltIn.Should Match Regexp ${node} ${NODE_READY_STATUS}
134 [Documentation] Create labels for minions so that random allocation of pods to minions is avoided
135 ${i} = BuiltIn.Set Variable 1
136 ${get nodes} = Utils.Run Command On Remote System And Log ${K8s_MASTER_IP} kubectl get nodes
137 @{get nodes} = String.Split To Lines ${get nodes} 2
138 : FOR ${status} IN @{get nodes}
139 \ ${minion} = BuiltIn.Should Match Regexp ${status} ^\\w+-.*-\\d+
140 \ Utils.Run Command On Remote System And Log ${K8s_MASTER_IP} kubectl label nodes ${minion} disktype=ss${i}
141 \ ${i} = BuiltIn.Evaluate ${i}+1
142 Utils.Run Command On Remote System And Log ${K8s_MASTER_IP} kubectl get nodes --show-labels
144 Derive Coe Data Models
145 [Documentation] Data models is created by integrating netvirt and coe data models which is given as input to get the model dumps
146 : FOR ${models} IN @{netvirt_data_models}
147 \ Collections.Append To List ${coe_data_models} ${models}
149 Check Pod Status Is Running
150 [Documentation] Checks the status of pods.This keyword is repeated until the status of all pods is Running
151 ${pods} = Utils.Run Command On Remote System ${K8s_MASTER_IP} kubectl get pods -o wide ${DEFAULT_USER} ${DEFAULT_PASSWORD} ${DEFAULT_LINUX_PROMPT_STRICT}
152 @{cluster} = String.Split To Lines ${pods} 1
153 : FOR ${pod} IN @{cluster}
154 \ BuiltIn.Should Match Regexp ${pod} ${POD_RUNNING_STATUS}
157 [Documentation] Test teardown to get dumpflows,ovsconfig,model dump,node status,pod status and to dump config files \ and delete pods.
158 : FOR ${conn_id} IN @{TOOLS_SYSTEM_ALL_CONN_IDS}
159 \ OVSDB.Get DumpFlows And Ovsconfig ${conn_id} ${bridge}
160 BuiltIn.Run Keyword And Ignore Error DataModels.Get Model Dump ${ODL_SYSTEM_IP} ${coe_data_models}
162 Utils.Run Command On Remote System And Log ${K8s_MASTER_IP} kubectl get nodes ${DEFAULT_USER} ${DEFAULT_PASSWORD} ${DEFAULT_LINUX_PROMPT_STRICT}
163 Utils.Run Command On Remote System And Log ${K8s_MASTER_IP} kubectl get pods -o wide ${DEFAULT_USER} ${DEFAULT_PASSWORD} ${DEFAULT_LINUX_PROMPT_STRICT}
167 [Documentation] Waits till the keyword delete status succeeds implying that all pods created have been deleted
168 ${get pods} = Utils.Run Command On Remote System ${K8s_MASTER_IP} kubectl get pods -o wide
169 @{get pods} = String.Split To Lines ${get pods} 1
170 : FOR ${status} IN @{get pods}
171 \ ${pod_name} = BuiltIn.Should Match Regexp ${status} ^\\w+-\\w+
172 \ Utils.Run Command On Remote System ${K8s_MASTER_IP} kubectl delete pods ${pod_name}
173 BuiltIn.Wait Until Keyword Succeeds 60s 3s Coe.Check If Pods Are Terminated
175 Check If Pods Are Terminated
176 [Documentation] Checks if the pods created have been terminated.The keyword is repeated until the pods are deleted
177 ${status} = Utils.Run Command On Remote System ${K8s_MASTER_IP} kubectl get pods -o wide ${DEFAULT_USER} ${DEFAULT_PASSWORD} ${DEFAULT_LINUX_PROMPT_STRICT}
178 ... ${DEFAULT_TIMEOUT} return_stdout=False return_stderr=True
179 BuiltIn.Should Contain ${status} No resources
182 [Documentation] Logs the configuration files present in all nodes
183 : FOR ${nodes} IN @{TOOLS_SYSTEM_ALL_IPS}
184 \ Utils.Run Command On Remote System And Log ${nodes} cat ${CONFIG_FILE}
187 [Documentation] Suite teardown keyword
188 Coe.Collect Watcher Log
189 Coe.Collect Journalctl Log
192 SSHLibrary.Close All Connections
195 [Documentation] Watcher running in background logs into watcher.out which is copied to ${JENKINS_WORKSPACE}/archives/watcher.log
196 SSHLibrary.Switch Connection ${TOOLS_SYSTEM_ALL_CONN_IDS[0]}
197 SSHLibrary.Get File /tmp/watcher.out ${JENKINS_WORKSPACE}/archives/watcher.log
199 Collect Journalctl Log
200 [Documentation] Logs of the command journalctl -u kubelet is copied to ${JENKINS_WORKSPACE}/archives/journal.log
201 Utils.Run Command On Remote System And Log ${K8s_MASTER_IP} sudo journalctl -u kubelet > ${USER_HOME}/journal.txt
202 SSHLibrary.Switch Connection ${TOOLS_SYSTEM_ALL_CONN_IDS[0]}
203 SSHLibrary.Get File ${USER_HOME}/journal.txt ${JENKINS_WORKSPACE}/archives/journalctl.log
206 [Documentation] Kill the watcher running at the background after completion of tests cases
207 ${watcher status} = Utils.Run Command On Remote System ${K8s_MASTER_IP} ps -ef | grep watcher
208 ${watcher} ${pid} = BuiltIn.Should Match Regexp ${watcher status} \\w+\\s+(\\d+).*watcher odl
209 Utils.Run Command On Remote System ${K8s_MASTER_IP} kill -9 ${pid}
212 [Documentation] Reset K8s to clear up all stale entries
213 : FOR ${nodes} IN @{NODE_IPs}
214 \ ${kube} = Utils.Run Command On Remote System And Log ${nodes} sudo kubeadm reset
215 \ BuiltIn.Should Contain ${kube} Stopping the kubelet service.
218 [Arguments] ${label} ${yaml} ${name}
219 [Documentation] Creates pods using the labels of the nodes and busy box names passed as arguments.
220 ${busybox} = OperatingSystem.Get File ${BUSY_BOX}
221 ${busybox} = String.Replace String ${busybox} string ${label}
222 ${busybox} = String.Replace String ${busybox} busyboxname ${name}
223 OperatingSystem.Create File ${VARIABLES_PATH}/${yaml} ${busybox}
224 SSHKeywords.Move_file_To_Remote_System ${K8s_MASTER_IP} ${VARIABLES_PATH}/${yaml} ${USER_HOME}
225 Utils.Run Command On Remote System And Log ${K8s_MASTER_IP} kubectl create -f ${yaml}
227 Collect Pod Names and Ping
228 [Documentation] This keyword collects the pod names and checks connectivity between each and every pod with respect to one another.
229 SSHLibrary.Switch Connection ${TOOLS_SYSTEM_ALL_CONN_IDS[0]}
230 ${get pods} = Write Commands Until Expected Prompt kubectl get pods -o wide ${DEFAULT_LINUX_PROMPT_STRICT}
231 @{pod ips} = String.Get Regexp Matches ${get pods} \\d+\\.\\d+\\.\\d+\\.\\d+
232 @{pod names} = String.Get Regexp Matches ${get pods} ss\\w+-\\w+
233 : FOR ${pod_name} IN @{pod names}
234 \ ${logs} = Log Statements ${pod ips} ${pod names} ${pod_name}
235 \ Ping Pods ${pod_name} ${pod ips} ${logs}
238 [Arguments] ${pod ips} ${pod names} ${pod_name}
239 @{log statement} = Create List
240 ${i} = Set Variable 0
241 : FOR ${pod_ip} IN @{pod ips}
242 \ ${ping statement} Set Variable Ping ${pod_name} and ${pod names[${i}]} : ${pod ip}
243 \ Append To List ${log statement} ${ping statement}
244 \ ${i} = Evaluate ${i}+1
245 [Return] @{log statement}
248 [Arguments] ${pod_name} ${pod ips} ${logs}
249 ${i} = Set Variable 0
250 : FOR ${ping info} IN @{logs}
251 \ ${ping} = Write Commands Until Expected Prompt kubectl exec -it ${pod_name} -- ping -c 3 ${pod ips[${i}]} ${DEFAULT_LINUX_PROMPT_STRICT}
252 \ BuiltIn.log ${ping}
253 \ Builtin.Should Match Regexp ${ping} ${PING_REGEXP}
254 \ ${i} Evaluate ${i}+1
257 [Documentation] COE project requires stop suite to be executed only for the last test suite.This keyword find the current suite,compares it with the stored last suite value and executes Coe.Stop suite only if the cuurent suite is equal to the last suite.
258 ${current suite} ${suite names updated} Extract current suite name
259 ${last_suite} = Set Variable ${suite names updated[-1]}
260 ${status} = BuiltIn.Evaluate '${last_suite}' == '${current suite}'
261 Run Keyword If '${status}' == 'True' Coe.Stop Suite
263 Extract current suite name
264 [Documentation] This keyword returns the name of current test suite.Appropriate replacement in text is done to make test suite names in SUITES and SUITE_NAME similar.
265 @{suite names} Get Regexp Matches ${SUITES} coe\\/(\\w+).robot 1
266 @{suite names updated} Create List
267 : FOR ${suites} IN @{suite names}
268 \ ${suites} Replace String ${suites} _ ${SPACE}
269 \ Append To List ${suite names updated} ${suites}
270 ${suite line} ${current suite} Should Match Regexp ${SUITE_NAME} .txt.(\\w.*)
271 [Return] ${current suite} ${suite names updated}