3 .. contents:: Table of Contents
6 **Prerequisites**: OpenDaylight requires Java 1.8.0 and Open vSwitch >= 2.5.0
8 Installing OpenDaylight on an existing OpenStack
9 ------------------------------------------------
10 * On the control host, `Download the latest OpenDaylight release <https://www.opendaylight.org/software/downloads>`_
11 * Uncompress it as root, and start OpenDaylight (you can start OpenDaylight
12 by running karaf directly, but exiting from the shell will shut it down):
16 tar xvfz distribution-karaf-0.5.1-Boron-SR1.tar.gz
17 cd distribution-karaf-0.5.1-Boron-SR1
18 ./bin/start # Start OpenDaylight as a server process
20 * Connect to the Karaf shell, and install the odl-netvirt-openstack bundle
21 and their dependencies:
25 ./bin/client # Connect to OpenDaylight with the client
26 opendaylight-user@root> feature:install odl-netvirt-openstack odl-mdsal-apidocs
28 Optional - Advanced OpenDaylight Installation - Configurations and Clustering
29 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
30 * ACL Implementation - Security Groups - Stateful:
32 * Default implementation used is stateful, requiring OVS compiled with conntrack modules.
33 * This requires using a linux kernel that is >= 4.3
34 * To check if OVS is running with conntrack support:
38 root@devstack:~/# lsmod | grep conntrack | grep openvswitch
39 nf_conntrack 106496 9 xt_CT,openvswitch,nf_nat,nf_nat_ipv4,xt_conntrack,nf_conntrack_netlink,xt_connmark,nf_conntrack_ipv4,nf_conntrack_ipv6
41 * If the conntrack modules are not installed for OVS, either recompile/install an OVS version with conntrack support, or alternatively configure OpenDaylight to use a non-stateful implementation.
42 * OpenvSwitch 2.5 with conntrack support can be acquired from this repository for yum based linux distributions:
46 yum install -y http://rdoproject.org/repos/openstack-newton/rdo-release-newton.rpm
47 yum install -y --nogpgcheck openvswitch
48 * To spwan a VM with security groups disabled:
52 crudini --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2 extension_drivers port_security
54 openstack port create --network=net1 --disable-port-security port1
55 openstack server create --flavor m1.tiny --image cirros --port port1 vm1
58 * Running multiple OpenDaylight controllers in a cluster:
60 * For redundancy, it is possible to run OpenDaylight in a 3-node cluster.
61 * More info on Clustering available `here <http://docs.opendaylight.org/en/latest/getting-started-guide/common-features/clustering.html>`_.
62 * To configure OpenDaylight in clustered mode, run <ODL_FOLDER>/bin/configure_cluster.sh on each node prior to running OpenDaylight.
63 This script is used to configure cluster parameters on this controller. The user should restart controller to apply changes.
67 Usage: ./configure_cluster.sh <index> <seed_nodes_list>
68 - index: Integer within 1..N, where N is the number of seed nodes.
69 - seed_nodes_list: List of seed nodes, separated by comma or space.
71 * The address at the provided index should belong this controller.
72 When running this script on multiple seed nodes, keep the seed_node_list same,
73 and vary the index from 1 through N.
75 * Optionally, shards can be configured in a more granular way by modifying the file
76 "custom_shard_configs.txt" in the same folder as this tool.
77 Please see that file for more details.
80 OpenDaylight should be restarted after applying any of the above changes via configuration files.
82 Ensuring OpenStack network state is clean
83 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
84 When using OpenDaylight as the Neutron back-end, OpenDaylight expects to be the only source of
85 truth for Neutron configurations. Because of this, it is necessary to remove existing OpenStack
86 configurations to give OpenDaylight a clean slate.
93 nova delete <instance names>
95 * Remove links from subnets to routers:
101 neutron router-port-list <router name>
102 neutron router-interface-delete <router name> <subnet ID or name>
104 * Delete subnets, networks, routers:
108 neutron subnet-delete <subnet name>
110 neutron net-delete <net name>
111 neutron router-delete <router name>
113 * Check that all ports have been cleared - at this point, this should be an
121 Ensure Neutron is stopped
122 ^^^^^^^^^^^^^^^^^^^^^^^^^
123 While Neutron is managing the OVS instances on compute and control nodes,
124 OpenDaylight and Neutron can be in conflict. To prevent issues, we turn off
125 Neutron server on the network controller, and Neutron's Open vSwitch agents
128 * Turn off neutron-server on control node:
132 systemctl stop neutron-server
133 systemctl stop neutron-l3-agent
135 * On each node in the cluster, shut down and disable Neutron's agent services to
136 ensure that they do not restart after a reboot:
140 systemctl stop neutron-openvswitch-agent
142 neutron-openvswitch-agent
143 systemctl stop neutron-l3-agent
144 systemctl disable neutron-l3-agent
147 Configuring Open vSwitch to be managed by OpenDaylight
148 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
149 On each host (both compute and control nodes) we will clear the pre-existing
150 Open vSwitch config and set OpenDaylight to manage the switch:
152 * Stop the Open vSwitch service, and clear existing OVSDB (OpenDaylight expects to
153 manage vSwitches completely):
157 systemctl stop openvswitch
158 rm -rf /var/log/openvswitch/*
159 rm -rf /etc/openvswitch/conf.db
160 systemctl start openvswitch
162 * At this stage, your Open vSwitch configuration should be empty:
166 [root@odl-compute2 ~]# ovs-vsctl show
167 9f3b38cb-eefc-4bc7-828b-084b1f66fbfd
170 * Set OpenDaylight as the manager on all nodes:
174 ovs-vsctl set-manager tcp:{CONTROL_HOST}:6640
176 * Set the IP to be used for VXLAN connectivity on all nodes.
177 This IP must correspond to an actual linux interface on each machine.
181 sudo ovs-vsctl set Open_vSwitch . other_config:local_ip=<ip>
183 * You should now see a new section in your Open vSwitch configuration
184 showing that you are connected to the OpenDaylight server via OVSDB,
185 and OpenDaylight will automatically create a br-int bridge that is
186 connected via OpenFlow to the controller:
189 :emphasize-lines: 4,6-7
191 [root@odl-compute2 ~]# ovs-vsctl show
192 9f3b38cb-eefc-4bc7-828b-084b1f66fbfd
193 Manager "tcp:172.16.21.56:6640"
196 Controller "tcp:172.16.21.56:6653"
203 [root@odl-compute2 ~]# ovs-vsctl get Open_vSwitch . other_config
204 {local_ip="10.0.42.161"}
206 * If you do not see the result above (specifically, if you do not see "is_connected: true" in the Manager section or in the Controller section), you may not have a security policies in place to allow Open vSwitch remote administration.
209 | There might be iptables restrictions - if so the relevant ports should be opened (6640, 6653).
210 | If SELinux is running on your linux, set to permissive mode on all nodes and ensure it stays that way after boot.
215 sed -i -e 's/SELINUX=enforcing/SELINUX=permissive/g' /etc/selinux/config
217 * Make sure all nodes, including the control node, are connected to OpenDaylight.
219 * If something has gone wrong, check ``data/log/karaf.log`` under
220 the OpenDaylight distribution directory. If you do not see any interesting
221 log entries, set logging for netvirt to TRACE level inside Karaf and try again:
225 log:set TRACE netvirt
227 Configuring Neutron to use OpenDaylight
228 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
229 Once you have configured the vSwitches to connect to OpenDaylight, you can
230 now ensure that OpenStack Neutron is using OpenDaylight.
232 This requires the neutron networking-odl module to be installed.
233 | ``pip install networking-odl``
235 First, ensure that port 8181 (which will be used by OpenDaylight to listen
236 for REST calls) is available. OpenDaylight can be configured to listen on a different port,
237 by modifying the ``jetty.port`` property value in ``etc/jetty.conf``.
242 <Property name="jetty.port" default="8181" />
245 * Configure Neutron to use OpenDaylight's ML2 driver:
249 crudini --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2 mechanism_drivers opendaylight
250 crudini --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2 tenant_network_types vxlan
252 cat <<EOT>> /etc/neutron/plugins/ml2/ml2_conf.ini
254 url = http://{CONTROL_HOST}:8181/controller/nb/v2/neutron
259 * Configure Neutron to use OpenDaylight's odl-router service plugin for L3 connectivity:
263 crudini --set /etc/neutron/plugins/neutron.conf DEFAULT service_plugins odl-router
265 * Configure Neutron DHCP agent to provide metadata services:
269 crudini --set /etc/neutron/plugins/dhcp_agent.ini DEFAULT force_metadata True
272 | If the OpenStack version being used is Newton, this workaround should be applied,
273 | configuring the Neutron DHCP agent to use vsctl as the OVSDB interface:
277 crudini --set /etc/neutron/plugins/dhcp_agent.ini OVS ovsdb_interface vsctl
279 * Reset Neutron's database
283 mysql -e "DROP DATABASE IF EXISTS neutron;"
284 mysql -e "CREATE DATABASE neutron CHARACTER SET utf8;"
285 /usr/local/bin/neutron-db-manage --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini upgrade head
287 * Restart neutron-server:
291 systemctl start neutron-server
295 * Verify that OpenDaylight's ML2 interface is working:
299 curl -u admin:admin http://{CONTROL_HOST}:8181/controller/nb/v2/neutron/networks
305 | If this does not work or gives an error, check Neutron's log file in ``/var/log/neutron/server.log``.
306 | Error messages here should give some clue as to what the problem is in the connection with OpenDaylight.
308 * Create a network, subnet, router, connect ports, and start an instance using the Neutron CLI:
312 neutron router-create router1
313 neutron net-create private
314 neutron subnet-create private --name=private_subnet 10.10.5.0/24
315 neutron router-interface-add router1 private_subnet
316 nova boot --flavor <flavor> --image <image id> --nic net-id=<network id> test1
317 nova boot --flavor <flavor> --image <image id> --nic net-id=<network id> test2
319 At this point, you have confirmed that OpenDaylight is creating network
320 end-points for instances on your network and managing traffic to them.
322 | VMs can be reached using Horizon console, or alternatively by issuing ``nova get-vnc-console <vm> novnc``
323 | Through the console, connectivity between VMs can be verified.
325 Adding an external network for floating IP connectivity
326 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
327 * In order to connect to the VM using a floating IP, we need to configure external network connectivity, by creating an external network and subnet. This external network must be linked to a physical port on the machine, which will provide connectivity to an external gateway.
331 sudo ovs-vsctl set Open_vSwitch . other_config:provider_mappings=physnet1:eth1
332 neutron net-create public-net -- --router:external --is-default --provider:network_type=flat --provider:physical_network=physnet1
333 neutron subnet-create --allocation-pool start=10.10.10.2,end=10.10.10.254 --gateway 10.10.10.1 --name public-subnet public-net 10.10.0.0/16 -- --enable_dhcp=False
334 neutron router-gateway-set router1 public-net
336 neutron floatingip-create public-net
337 nova floating-ip-associate test1 <floating_ip>
339 Installing OpenStack and OpenDaylight using DevStack
340 ----------------------------------------------------
341 The easiest way to load and OpenStack setup using OpenDaylight is by using devstack, which does all the steps mentioned in previous sections.
342 | ``git clone https://git.openstack.org/openstack-dev/devstack``
344 * The following lines need to be added to your local.conf:
348 enable_plugin networking-odl http://git.openstack.org/openstack/networking-odl <branch>
350 Q_ML2_PLUGIN_MECHANISM_DRIVERS=opendaylight,logger
351 ODL_GATE_SERVICE_PROVIDER=vpnservice
353 ML2_L3_PLUGIN=odl-router
354 ODL_PROVIDER_MAPPINGS={PUBLIC_PHYSICAL_NETWORK}:<external linux interface>
356 * More details on using devstack can be found in the following links:
358 * `Devstack All-In-One Single Machine Tutorial <http://docs.openstack.org/developer/devstack/guides/single-machine.html>`_
359 * `Devstack networking-odl README <https://github.com/openstack/networking-odl/blob/master/devstack/README.rst>`_
365 * Trigger DHCP requests - access VM console:
367 * View log: ``nova console-log <vm>``
368 * Access using VNC console: nova get-vnc-console <vm> novnc
369 * Trigger DHCP requests:
370 ``sudo ifdown eth0 ; sudo ifup eth0``
374 udhcpc (v1.20.1) started
376 Sending select for 10.0.123.3...
377 Lease of 10.0.123.3 obtained, lease time 86400 # This only happens when DHCP is properly obtained.
379 * Check if the DHCP requests are reaching the qdhcp agent using the following commands on the OpenStack controller:
384 sudo ip netns exec qdhcp-xxxxx ifconfig # xxxx is the neutron network id
385 sudo ip netns exec qdhcp-xxxxx tcpdump -nei tapxxxxx # xxxxx is the neutron port id
387 # Valid request and response:
388 15:08:41.684932 fa:16:3e:02:14:bb > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 329: 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from fa:16:3e:02:14:bb, length 287
389 15:08:41.685152 fa:16:3e:79:07:98 > fa:16:3e:02:14:bb, ethertype IPv4 (0x0800), length 354: 10.0.123.2.67 > 10.0.123.3.68: BOOTP/DHCP, Reply, length 312
391 * If the requests aren't reaching qdhcp:
393 * Verify VXLAN tunnels exist between compute and control nodes by using ``ovs-vsctl show``
394 * | Run the following commands to debug the OVS processing of the DHCP request packet:
395 | ``ovs-ofctl -OOpenFlow13 dump-ports-desc br-int`` # retrieve VMs ofport and MAC
396 | ``ovs-appctl ofproto/trace br-int in_port=<ofport>,dl_src=<mac>,dl_dst=ff:ff:ff:ff:ff:ff,udp,ip_src=0.0.0.0,ip_dst=255.255.255.255 | grep "Rule\|action"``
400 root@devstack:~# ovs-appctl ofproto/trace br-int in_port=1,dl_src=fe:16:3e:33:8b:d8,dl_dst=ff:ff:ff:ff:ff:ff,udp,ip_src=0.0.0.0,ip_dst=255.255.255.255 | grep "Rule\|action"
401 Rule: table=0 cookie=0x8000000 priority=1,in_port=1
402 OpenFlow actions=write_metadata:0x20000000001/0xffffff0000000001,goto_table:17
403 Rule: table=17 cookie=0x8000001 priority=5,metadata=0x20000000000/0xffffff0000000000
404 OpenFlow actions=write_metadata:0xc0000200000222e2/0xfffffffffffffffe,goto_table:19
405 Rule: table=19 cookie=0x1080000 priority=0
406 OpenFlow actions=resubmit(,17)
407 Rule: table=17 cookie=0x8040000 priority=6,metadata=0xc000020000000000/0xffffff0000000000
408 OpenFlow actions=write_metadata:0xe00002138a000000/0xfffffffffffffffe,goto_table:50
409 Rule: table=50 cookie=0x8050000 priority=0
410 OpenFlow actions=CONTROLLER:65535,goto_table:51
411 Rule: table=51 cookie=0x8030000 priority=0
412 OpenFlow actions=goto_table:52
413 Rule: table=52 cookie=0x870138a priority=5,metadata=0x138a000001/0xffff000001
414 OpenFlow actions=write_actions(group:210003)
415 Datapath actions: drop
417 root@devstack:~# ovs-ofctl -OOpenFlow13 dump-groups br-int | grep 'group_id=210003'
418 group_id=210003,type=all
420 * If the requests are reaching qdhcp, but the response isn't arriving to the VM:
422 * Locate the compute the VM is residing on (can use ``nova show <vm>``).
424 * | If the VM is on the same node as the qdhcp namespace, ``ofproto/trace`` can be used to track the packet:
425 | ``ovs-appctl ofproto/trace br-int in_port=<dhcp_ofport>,dl_src=<dhcp_port_mac>,dl_dst=<vm_port_mac>,udp,ip_src=<dhcp_port_ip>,ip_dst=<vm_port_ip> | grep "Rule\|action"``
429 root@devstack:~# ovs-appctl ofproto/trace br-int in_port=2,dl_src=fa:16:3e:79:07:98,dl_dst=fa:16:3e:02:14:bb,udp,ip_src=10.0.123.2,ip_dst=10.0.123.3 | grep "Rule\|action"
430 Rule: table=0 cookie=0x8000000 priority=4,in_port=2
431 OpenFlow actions=write_metadata:0x10000000000/0xffffff0000000001,goto_table:17
432 Rule: table=17 cookie=0x8000001 priority=5,metadata=0x10000000000/0xffffff0000000000
433 OpenFlow actions=write_metadata:0x60000100000222e0/0xfffffffffffffffe,goto_table:19
434 Rule: table=19 cookie=0x1080000 priority=0
435 OpenFlow actions=resubmit(,17)
436 Rule: table=17 cookie=0x8040000 priority=6,metadata=0x6000010000000000/0xffffff0000000000
437 OpenFlow actions=write_metadata:0x7000011389000000/0xfffffffffffffffe,goto_table:50
438 Rule: table=50 cookie=0x8051389 priority=20,metadata=0x11389000000/0xfffffffff000000,dl_src=fa:16:3e:79:07:98
439 OpenFlow actions=goto_table:51
440 Rule: table=51 cookie=0x8031389 priority=20,metadata=0x1389000000/0xffff000000,dl_dst=fa:16:3e:02:14:bb
441 OpenFlow actions=load:0x300->NXM_NX_REG6[],resubmit(,220)
442 Rule: table=220 cookie=0x8000007 priority=7,reg6=0x300
443 OpenFlow actions=output:3
445 * If the VM isn't on the same node as the qdhcp namepsace:
447 * Check if the packet is arriving via VXLAN by running ``tcpdump -nei <vxlan_port> port 4789``
448 * If it is arriving via VXLAN, the packet can be tracked on the compute node rules, using ``ofproto/trace``
449 in a similiar manner to the previous section. Note that packets arriving from a tunnels have a unique
450 tunnel_id (VNI) that should be used as well in the trace, due to the special processing of packets arriving
455 * If you have assigned an external network and associated a floating IP to a VM but there is still no connectivity:
457 * Verify the external gateway IP is reachable through the provided provider network port.
458 * Verify OpenDaylight has successfully resolved the MAC address of the external gateway IP.
459 This can be verified by searching for the line "Installing ext-net group" in the karaf.log.
460 * Locate the compute the VM is residing on (can use ``nova show <vm>``).
461 * Run a ping to the VM floating IP.
462 * If the ping fails, execute a flow dump of br-int, and search for the flows that are relevant to the VM's floating IP address:
463 ``ovs-ofctl -OOpenFlow13 dump-flows br-int | grep "<floating_ip>"``
465 * | Are there packets on the incoming flow (matching dst_ip=<floating_ip>)?
466 | If not this probably means the provider network has not been set up properly, verify provider_mappings configuration and the configured external network physical_network value match. Also verify that the Flat/VLAN network configured is actually reachable via the configured port.
467 * | Are there packets on the outgoing flow (matching src_ip=<floating_ip>)?
468 | If not, this probably means that OpenDaylight is failing to resolve the MAC of the provided external gateway, required for forwarding packets to the external network.
469 * | Are there packets being sent on the external network port?
470 | This can be checked using ``tcpdump <port>`` or by viewing the appropriate OpenFlow rules. The mapping between the OpenFlow port number and the linux interface can be acquired using ``ovs-ofctl dump-ports-desc br-int``
474 ovs-ofctl -OOpenFlow13 dump-flows br-int | grep "<floating_ip>"
475 cookie=0x8000003, duration=436.710s, table=21, n_packets=190, n_bytes=22602, priority=42,ip,metadata=0x222e2/0xfffffffe,nw_dst=10.64.98.17 actions=goto_table:25
476 cookie=0x8000004, duration=436.739s, table=25, n_packets=190, n_bytes=22602, priority=10,ip,nw_dst=10.64.98.17 actions=set_field:10.0.123.3->ip_dst,write_metadata:0x222e0/0xfffffffe,goto_table:27
477 cookie=0x8000004, duration=436.730s, table=26, n_packets=120, n_bytes=15960, priority=10,ip,metadata=0x222e0/0xfffffffe,nw_src=10.0.123.3 actions=set_field:10.64.98.17->ip_src,write_metadata:0x222e2/0xfffffffe,goto_table:28
478 cookie=0x8000004, duration=436.728s, table=28, n_packets=120, n_bytes=15960, priority=10,ip,metadata=0x222e2/0xfffffffe,nw_src=10.64.98.17 actions=set_field:fa:16:3e:ec:a8:84->eth_src,group:200000
482 * `NetVirt Tables Pipeline <https://docs.google.com/presentation/d/15h4ZjPxblI5Pz9VWIYnzfyRcQrXYxA1uUoqJsgA53KM>`_
483 * `NetVirt Wiki Page <https://wiki.opendaylight.org/view/NetVirt>`_
484 * `NetVirt Basic Tutorial (OpenDaylight Summit 2016) <https://docs.google.com/presentation/d/1VLzRIOEptSOY1b0w4PezRIQ0gF5vx7GyLKECWXRV5mE>`_
485 * `NetVirt Advanced Tutorial (OpenDaylight Summit 2016) <https://docs.google.com/presentation/d/13K8Z1kl5XFZrWqBToMwFISSAPOKfzd3m9BtVcb-YAWs>`_
486 * `Other OpenDaylight Documentation <http://docs.opendaylight.org/>`_