5 Resource DataModels.robot
8 Resource SSHKeywords.robot
10 Resource ../variables/netvirt/Variables.robot
11 Resource ../variables/Variables.robot
12 Resource VpnOperations.robot
13 Variables ../variables/coe/Modules.py
14 Variables ../variables/netvirt/Modules.py
17 ${CNI_BINARY_FILE} /opt/cni/bin/odlovs-cni
18 ${CONFIG_FILE} /etc/cni/net.d/odlovs-cni.conf
19 ${CONFIG_FILE_TEMPLATE} ${CURDIR}/../variables/coe/odlovs-cni.conf.j2
20 ${HOST_INVENTORY} ${CURDIR}/../variables/coe/hosts.yaml
21 ${K8s_MASTER_IP} ${TOOLS_SYSTEM_1_IP}
22 ${K8s_MINION1_IP} ${TOOLS_SYSTEM_2_IP}
23 ${K8s_MINION2_IP} ${TOOLS_SYSTEM_3_IP}
24 ${NODE_READY_STATUS} \\sReady
25 ${PLAYBOOK} ${CURDIR}/../variables/coe/coe_play.yaml
26 ${POD_RUNNING_STATUS} \\sRunning
27 ${WATCHER_COE} ${CURDIR}/../variables/coe/coe.yaml
28 @{NODE_IPs} ${K8s_MASTER_IP} ${K8s_MINION1_IP} ${K8s_MINION2_IP}
29 @{COE_DIAG_SERVICES} OPENFLOW IFM ITM DATASTORE ELAN OVSDB
33 [Documentation] Suite setup keyword.
34 Coe.Configuration Playbook
35 Coe.Set Connection ids and Bridge
36 Coe.Verify Config Files
37 Coe.Verify Watcher Is Running
38 BuiltIn.Wait Until Keyword Succeeds 40s 2s Coe.Check Node Status Is Ready
40 BuiltIn.Wait Until Keyword Succeeds 60 2 ClusterManagement.Check Status Of Services Is OPERATIONAL @{COE_DIAG_SERVICES}
41 Genius.Verify Tunnel Status as UP default-transport-zone
42 Coe.Derive Coe Data Models
44 Configuration Playbook
45 [Documentation] Ansible playbook which does all basic configuration for kubernetes nodes.
46 ${hosts} = OperatingSystem.Get File ${HOST_INVENTORY}
47 ${hosts} = String.Replace String ${hosts} master_ip ${K8s_MASTER_IP}
48 ${hosts} = String.Replace String ${hosts} minion1_ip ${K8s_MINION1_IP}
49 ${hosts} = String.Replace String ${hosts} minion2_ip ${K8s_MINION2_IP}
50 ${hosts} = String.Replace String ${hosts} odl_ip ${ODL_SYSTEM_IP}
51 ${hosts} = String.Replace String ${hosts} mport ${OVSDBPORT}
52 ${hosts} = String.Replace String ${hosts} cport ${ODL_OF_PORT_6653}
53 ${hosts} = String.Replace String ${hosts} filepath ${CONFIG_FILE_TEMPLATE}
54 ${hosts} = String.Replace String ${hosts} yamlpath ${USER_HOME}/coe.yaml
55 OperatingSystem.Create File ${USER_HOME}/hosts.yaml ${hosts}
56 ${watcher} = OperatingSystem.Get File ${WATCHER_COE}
57 ${watcher} = String.Replace String ${watcher} odlip ${ODL_SYSTEM_IP}
58 ${watcher} = String.Replace String ${watcher} port ${RESTCONFPORT}
59 OperatingSystem.Create File ${WATCHER_COE} ${watcher}
60 SSHKeywords.Copy_File_To_Remote_System ${K8s_MASTER_IP} ${WATCHER_COE} ${USER_HOME}
61 OperatingSystem.Move File ${PLAYBOOK} ${USER_HOME}
62 ${play_output} = OperatingSystem.Run ansible-playbook ${USER_HOME}/coe_play.yaml -i ${USER_HOME}/hosts.yaml
63 BuiltIn.Log ${play_output}
65 Set Connection ids and Bridge
66 [Documentation] Sets the connection ids for all the nodes and get the bridge from configuration file .
67 ${conn_id_1} = SSHLibrary.Open Connection ${K8s_MASTER_IP}
68 SSHKeywords.Flexible_SSH_Login ${DEFAULT_USER} ${DEFAULT_PASSWORD}
69 BuiltIn.Set Suite Variable ${conn_id_1}
70 ${conn_id_2} = SSHLibrary.Open Connection ${K8s_MINION1_IP}
71 SSHKeywords.Flexible_SSH_Login ${DEFAULT_USER} ${DEFAULT_PASSWORD}
72 BuiltIn.Set Suite Variable ${conn_id_2}
73 ${conn_id_3} = SSHLibrary.Open Connection ${K8s_MINION2_IP}
74 SSHKeywords.Flexible_SSH_Login ${DEFAULT_USER} ${DEFAULT_PASSWORD}
75 BuiltIn.Set Suite Variable ${conn_id_3}
76 ${file} = OperatingSystem.Get File ${CONFIG_FILE_TEMPLATE}
77 ${line} ${bridge} = Should Match Regexp ${file} "ovsBridge": "(\\w.*)"
78 BuiltIn.Set Suite Variable ${bridge}
81 [Documentation] Checks if the configuration files are present in all nodes
82 : FOR ${nodes} IN @{NODE_IPs}
83 \ Utils.Verify File Exists On Remote System ${nodes} ${CONFIG_FILE}
84 : FOR ${nodes} IN @{NODE_IPs}
85 \ Utils.Verify File Exists On Remote System ${nodes} ${CNI_BINARY_FILE}
87 Verify Watcher Is Running
88 [Documentation] Checks if watcher is running in the background
89 ${lines} = Utils.Run Command On Remote System ${K8s_MASTER_IP} ps -ef | grep watcher
90 BuiltIn.Should Match Regexp ${lines} .* watcher odl
92 Check Node Status Is Ready
93 [Documentation] Checks the status of nodes.This keyword is repeated until the status of all nodes is Ready
94 ${nodes} = Utils.Run Command On Remote System ${K8s_MASTER_IP} kubectl get nodes ${DEFAULT_USER} ${DEFAULT_PASSWORD} ${DEFAULT_LINUX_PROMPT_STRICT}
95 @{cluster} = String.Split To Lines ${nodes} 1
96 : FOR ${node} IN @{cluster}
97 \ BuiltIn.Should Match Regexp ${node} ${NODE_READY_STATUS}
100 [Documentation] Create labels for minions so that random allocation of pods to minions is avoided
101 ${nodes} = Utils.Run Command On Remote System ${K8s_MASTER_IP} kubectl get nodes
102 ${node_1} = String.Get Line ${nodes} 2
103 ${minion_1} = BuiltIn.Should Match Regexp ${node_1} ^\\w+-.*-\\d+
104 ${node_2} = String.Get Line ${nodes} 3
105 ${minion_2} = BuiltIn.Should Match Regexp ${node_2} ^\\w+-.*-\\d+
106 Utils.Run Command On Remote System And Log ${K8s_MASTER_IP} kubectl label nodes ${minion_1} disktype=ssd
107 Utils.Run Command On Remote System And Log ${K8s_MASTER_IP} kubectl label nodes ${minion_2} disktype=ssl
108 Utils.Run Command On Remote System And Log ${K8s_MASTER_IP} kubectl get nodes --show-labels
110 Derive Coe Data Models
111 [Documentation] Data models is created by integrating netvirt and coe data models which is given as input to get the model dumps
112 : FOR ${models} IN @{netvirt_data_models}
113 \ Collections.Append To List ${coe_data_models} ${models}
115 Check Pod Status Is Running
116 [Documentation] Checks the status of pods.This keyword is repeated until the status of all pods is Running
117 ${pods} = Utils.Run Command On Remote System ${K8s_MASTER_IP} kubectl get pods -o wide ${DEFAULT_USER} ${DEFAULT_PASSWORD} ${DEFAULT_LINUX_PROMPT_STRICT}
118 @{cluster} = String.Split To Lines ${pods} 1
119 : FOR ${pod} IN @{cluster}
120 \ BuiltIn.Should Match Regexp ${pod} ${POD_RUNNING_STATUS}
123 [Documentation] Test teardown to get dumpflows,ovsconfig,model dump,node status,pod status and to dump config files \ and delete pods.
124 OVSDB.Get DumpFlows And Ovsconfig ${conn_id_1} ${bridge}
125 OVSDB.Get DumpFlows And Ovsconfig ${conn_id_2} ${bridge}
126 OVSDB.Get DumpFlows And Ovsconfig ${conn_id_3} ${bridge}
127 BuiltIn.Run Keyword And Ignore Error DataModels.Get Model Dump ${ODL_SYSTEM_IP} ${coe_data_models}
129 Utils.Run Command On Remote System And Log ${K8s_MASTER_IP} kubectl get nodes ${DEFAULT_USER} ${DEFAULT_PASSWORD} ${DEFAULT_LINUX_PROMPT_STRICT}
130 Utils.Run Command On Remote System And Log ${K8s_MASTER_IP} kubectl get pods -o wide ${DEFAULT_USER} ${DEFAULT_PASSWORD} ${DEFAULT_LINUX_PROMPT_STRICT}
134 [Documentation] Waits till the keyword delete status succeeds implying that all pods created have been deleted
135 ${lines} = Utils.Run Command On Remote System ${K8s_MASTER_IP} kubectl get pods -o wide
136 @{lines} = String.Split To Lines ${lines} 1
137 : FOR ${status} IN @{lines}
138 \ ${pod_name} = BuiltIn.Should Match Regexp ${status} ^\\w+
139 \ Utils.Run Command On Remote System ${K8s_MASTER_IP} kubectl delete pods ${pod_name}
140 BuiltIn.Wait Until Keyword Succeeds 60s 3s Coe.Check If Pods Are Terminated
142 Check If Pods Are Terminated
143 [Documentation] Checks if the pods created have been terminated.The keyword is repeated until the pods are deleted
144 ${status} = Utils.Run Command On Remote System ${K8s_MASTER_IP} kubectl get pods -o wide ${DEFAULT_USER} ${DEFAULT_PASSWORD} ${DEFAULT_LINUX_PROMPT_STRICT}
145 ... ${DEFAULT_TIMEOUT} return_stdout=False return_stderr=True
146 BuiltIn.Should Contain ${status} No resources
149 [Documentation] Logs the configuration files present in all nodes
150 : FOR ${nodes} IN @{NODE_IPs}
151 \ Utils.Run Command On Remote System And Log ${nodes} cat ${CONFIG_FILE}
154 [Documentation] Suite teardown keyword
155 Coe.Collect Watcher Log
156 Coe.Collect Journalctl Log
159 SSHLibrary.Close All Connections
162 [Documentation] Watcher running in background logs into watcher.out which is copied to ${JENKINS_WORKSPACE}/archives/watcher.log
163 SSHLibrary.Open Connection ${K8s_MASTER_IP}
164 SSHKeywords.Flexible_SSH_Login ${DEFAULT_USER} ${DEFAULT_PASSWORD}
165 SSHLibrary.Get File /tmp/watcher.out ${JENKINS_WORKSPACE}/archives/watcher.log
166 SSHLibrary.Close Connection
168 Collect Journalctl Log
169 [Documentation] Logs of the command journalctl -u kubelet is copied to ${JENKINS_WORKSPACE}/archives/journal.log
170 Utils.Run Command On Remote System And Log ${K8s_MASTER_IP} sudo journalctl -u kubelet > ${USER_HOME}/journal.txt
171 SSHLibrary.Open Connection ${K8s_MASTER_IP}
172 SSHKeywords.Flexible_SSH_Login ${DEFAULT_USER} ${DEFAULT_PASSWORD}
173 SSHLibrary.Get File ${USER_HOME}/journal.txt ${JENKINS_WORKSPACE}/archives/journalctl.log
174 SSHLibrary.Close Connection
177 [Documentation] Kill the watcher running at the background after completion of tests cases
178 ${lines} = Utils.Run Command On Remote System ${K8s_MASTER_IP} ps -ef | grep watcher
179 ${line} ${pid} = BuiltIn.Should Match Regexp ${lines} \\w+\\s+(\\d+).*watcher odl
180 Utils.Run Command On Remote System ${K8s_MASTER_IP} kill -9 ${pid}
183 [Documentation] Reset K8s to clear up all stale entries
184 : FOR ${nodes} IN @{NODE_IPs}
185 \ ${kube} = Utils.Run Command On Remote System And Log ${nodes} sudo kubeadm reset
186 \ BuiltIn.Should Contain ${kube} Stopping the kubelet service.