1 .. contents:: Table of Contents
4 =========================================================
5 Support of VXLAN based L2 connectivity across Datacenters
6 =========================================================
8 https://git.opendaylight.org/gerrit/#/q/topic:EVPN_RT2
10 Enable realization of L2 connectivity over VXLAN tunnels using L2 BGPVPNs,
11 internally taking advantage of EVPN as the BGP Control Plane mechanism.
16 OpenDaylight NetVirt service today supports L3VPN connectivity over VXLAN tunnels.
17 L2DCI communication is not possible so far.
19 This spec attempts to enhance the BGPVPN service in NetVirt to
20 embrace inter-DC L2 connectivity over external VXLAN tunnels.
25 The scope primarily includes providing ability to support intra-subnet
26 connectivity across DataCenters over VXLAN tunnels using BGP EVPN with type L2.
28 When we mention that we are using EVPN BGP Control plane, this
29 spec proposes using the RouteType 2 as the primary
30 means to provision the control plane to enable inter-DC connectivity
31 over external VXLAN tunnels.
33 With this inplace we will be able to support the following.
35 * Intra-subnet connectivity across dataCenters over VXLAN tunnels.
37 The following are already supported as part of the other spec(RT5)
38 and will continue to function.
40 * Intra-subnet connectivity within a DataCenter over VXLAN tunnels.
41 * Inter-subnet connectivity within a DataCenter over VXLAN tunnels.
42 * Inter-subnet connectivity across dataCenters over VXLAN tunnels.
50 The following high level use-cases will be realized by the implementation of this Spec.
52 Datacenter access from another Datacenter over WAN via respective DC-Gateways (L2 DCI)
53 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
55 This use-case involves providing intra-subnet connectivity between two DataCenters.
56 Tenant VMs in one datacenter will be able to communicate with tenant VMs on the other
57 datacenter provided they are part of the same BGP EVPN and they are on same subnets.
59 The dataplane between the tenant VMs themselves and between the tenant VMs
60 towards the DC-Gateway will be over VXLAN Tunnels.
62 The dataplane between the DC-Gateway to its other WAN-based BGP Peers is
63 transparent to this spec. It is usually MPLS-based EPVPN.
65 The BGP Control plane between the ODL Controller and the DC-Gateway will be
66 via EVPN RouteType 2 as defined in EVPN_RT2.
68 The control plane between the DC-Gateway and it other BGP Peers in the WAN
69 is transparent to this spec, but can be EVPN IP-MPLS.
73 1. We will have only a single DCGW for WAN connectivity
74 2. MAC IP prefix exchange between ODL controller and DC-GW (iBGP) using EVPN RT2
75 3. WAN control plane may use EVPN IP-MPLS for route exchange.
76 4. On the DC-Gateway, the VRF instance will be configured with two sets of import/export targets. One set of import/export route targets belong to EVPN inside DataCenter (realized using EVPN RT2) and the second set of import/export route target belongs to WAN control plane.
77 5. EVPN single homing to be used in all RT2 exchanges inside the DataCenter i.e., ESI=0 for all prefixes sent from DataCenter to the DC-Gateway.
83 The following components of an Openstack-ODL-based solution need to be enhanced to provide
84 intra-subnet and inter-subnet connectivity across DCs using EVPN MAC IP Advertisement
85 (Route Type 2) mechanism (refer EVPN_RT2):
87 * Openstack Neutron BGPVPN Driver
88 * OpenDaylight Controller (NetVirt)
89 * BGP Quagga Stack to support EVPN with RouteType 2 NLRI
90 * DC-Gateway BGP Neighbour that supports EVPN with RouteType 2 NLRI
92 The changes required in Openstack Neutron BGPVPN Driver and BGP Quagga Stack
93 are captured in the Solution considerations section down below.
101 Intra Subnet, Local DPN: VMs on the same subnet, same VPN, same DPN
102 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
104 There are no explicit pipeline changes for this use-case.
106 Intra Subnet, Remote DPN: VMs on two different DPNs, both VMs on the same subnet and same VPN
107 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
109 There are no explicit pipeline changes for this use-case.
111 Inter Subnet, Local DPN: VMs on different subnet, same VPN, same DPN
112 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
114 There are no explicit pipeline changes for this use-case.
116 Inter Subnet, Local DPN: VMs on different subnet, same VPN, same DPN
117 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
119 There are no explicit pipeline changes for this use-case.
124 Intra subnet Traffic from DC-Gateway to Local DPN
125 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
127 | Classifier table (0) =>
128 | Dispatcher table (17) ``match: tunnel-type=vxlan`` =>
129 | L2VNI_EXTERNAL_TUNNEL_DEMUX_TABLE (24) => ``match tunnel-id=l2vni, set elan-tag``
130 | ELAN DMAC table (51) ``match: elan-tag=vxlan-net-tag,dst-mac=vm2-mac set reg6=vm-lport-tag`` =>
131 | Egress table (220) ``match: reg6=vm-lport-tag output to vm port``
133 Intra subnet Traffic from Local DPN to DC-Gateway
134 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
136 | Classifier table (0) =>
137 | Dispatcher table (17) ``l3vpn service: set vpn-id=router-id`` =>
138 | GW Mac table (19) =>
139 | Dispatcher table (17) ``l2vpn service: set elan-tag=vxlan-net-tag`` =>
140 | ELAN base table (48) =>
141 | ELAN SMAC table (50) ``match: elan-tag=vxlan-net-tag,src-mac=vm1-mac`` =>
142 | ELAN DMAC table (51) ``match: elan-tag=vxlan-net-tag,dst-mac=external-vm-mac set tun-id=vxlan-net-tag group=next-hop-group``
143 | Next Hop Group ``bucket0 :set reg6=tunnel-lport-tag bucket1 :set reg6=tunnel2-lport-tag``
144 | Egress table (220) ``match: reg6=tunnel2-lport-tag`` output to ``tunnel2``
147 Inter subnet Traffic from Local DPN to DC-Gateway ( Symmetric IRB )
148 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
150 | Classifier Table (0) =>
151 | Dispatcher table (17) ``l3vpn service: set vpn-id=router-id`` =>
152 | L3 Gateway MAC Table (19) ``match: vpn-id=l3vpn-id, dst-mac=vpn-subnet-gateway-mac-address`` =>
153 | L3 FIB Table (21) ``match: vpn-id=l3vpn-id, nw-dst=dst-vm-ip-address set tun-id=l3vni output to nexthopgroup`` =>
154 | NextHopGroup: ``set-eth-dst router-gw-vm, reg6=tunnel-lport-tag`` =>
155 | Lport Egress Table (220) ``Output to tunnel port``
157 Inter subnet Traffic from DC-Gateway to Local DPN ( Symmetric IRB )
158 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
160 | Classifier table (0) =>
161 | Dispatcher table (17) ``match: tunnel-type=vxlan`` =>
162 | L3VNI_EXTERNAL_TUNNEL_DEMUX_TABLE (23) => ``match tunnel-id=l3vni, set l3vpn-id`` =>
163 | L3 Gateway MAC Table (19) => ``match dst-mac=vpn-subnet-gateway-mac-address`` =>
164 | FIB table (21) ``match: l3vpn-tag=l3vpn-id,dst-ip=vm2-ip set reg6=vm-lport-tag goto=local-nexthop-group`` =>
165 | local nexthop group ``set dst-mac=vm2-mac table=220`` =>
166 | Egress table (220) ``match: reg6=vm-lport-tag output to vm port``
168 Inter subnet Traffic from Local DPN to DC-Gateway ( ASymmetric IRB )
169 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
171 | Classifier Table (0) =>
172 | Dispatcher table (17) ``l3vpn service: set vpn-id=router-id`` =>
173 | L3 Gateway MAC Table (19) ``match: vpn-id=l3vpn-id, dst-mac=vpn-subnet-gateway-mac-address`` =>
174 | L3 FIB Table (21) ``match: vpn-id=l3vpn-id, nw-dst=dst-vm-ip-address set tun-id=l2vni output to nexthopgroup`` =>
175 | NextHopGroup: ``set-eth-dst dst-vm-mac, reg6=tunnel-lport-tag`` =>
176 | Lport Egress Table (220) ``Output to tunnel port``
178 Intra subnet Traffic from DC-Gateway to Local DPN ( ASymmetric IRB )
179 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
181 | Classifier table (0) =>
182 | Dispatcher table (17) ``match: tunnel-type=vxlan`` =>
183 | L2VNI_EXTERNAL_TUNNEL_DEMUX_TABLE (24) => ``match tunnel-id=l2vni, set elan-tag``
184 | ELAN DMAC table (51) ``match: elan-tag=vxlan-net-tag,dst-mac=vm2-mac set reg6=vm-lport-tag`` =>
185 | Egress table (220) ``match: reg6=vm-lport-tag output to vm port``
189 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
191 Local DPN: VMs on the same subnet, same DPN
192 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
193 a. Introducing a new Table aka ELAN_ARP_SERVICE_TABLE (Table 81).
194 This table will be the first table in elan pipeline.
196 | Classifier table (0) =>
197 | Dispatcher table (17) ``elan service: set elan-id=vxlan-net-tag`` =>
198 | Arp Service table (81) => ``match: arp-op=req, dst-ip=vm-ip, ela-id=vxlan-net-tag inline arp reply``
200 Intra Subnet, Local DPN: VMs on the same subnet, on remote DC
201 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
203 | Classifier table (0) =>
204 | Dispatcher table (17) ``elan service: set elan-id=vxlan-net-tag`` =>
205 | Arp Service table (81) => ``match: arp-op=req, dst-ip=vm-ip, ela-id=vxlan-net-tag inline arp reply``
210 Changes will be needed in ``l3vpn.yang`` , ``odl-l3vpn.yang`` , ``odl-fib.yang`` and
211 ``neutronvpn.yang`` to start supporting EVPN functionality.
213 ODL-L3VPN YANG changes
214 ++++++++++++++++++++++
215 A new container evpn-rd-to-networks is added
216 This holds the rd to networks mapping
217 This will be useful to extract in which elan the received RT2 route can be injected into.
220 :caption: odl-l3vpn.yang
222 container evpn-rd-to-networks {
224 description "Holds the networks to which given evpn is attached to";
225 list evpn-rd-to-network {
241 A new field macVrfEntries is added to the container ``fibEntries``
242 This holds the RT2 routes received for the given rd
245 :caption: odl-fib.yang
247 grouping vrfEntryBase {
262 description "MPLSOverGRE";
275 key "nexthop-address";
276 leaf nexthop-address {
282 leaf gateway_mac_address {
296 grouping macVrfEntries{
306 container fibEntries {
309 key "routeDistinguisher";
310 leaf routeDistinguisher {type string;}
312 uses macVrfEntries;//new field
319 NEUTRONVPN YANG changes
320 +++++++++++++++++++++++
321 A new rpc ``createEVPN`` is added
322 Existing rpc associateNetworks is reused to attach a network to EVPN assuming
323 uuid of L3VPN and EVPN does not collide with each other.
326 :caption: neutronvpn.yang
329 description "Create one or more EVPN(s)";
338 description "Status response for createVPN RPC";
344 description "delete EVPNs for specified Id list";
348 description "evpn-id";
354 description "Status response for deleteEVPN RPC";
359 grouping evpn-instance {
364 description "evpn-id";
369 description "EVPN name";
374 description "The UUID of the tenant that will own the subnet.";
377 leaf-list route-distinguisher {
380 "configures a route distinguisher (RD) for the EVPN instance.
381 Format is ASN:nn or IP-address:nn.";
384 leaf-list import-RT {
387 "configures a list of import route target.
388 Format is ASN:nn or IP-address:nn.";
394 "configures a list of export route targets.
395 Format is ASN:nn or IP-address:nn.";
404 +++++++++++++++++++++++
405 Existing container elan-instances is augmented with evpn information.
407 A new list ``external-teps`` is added to elan container.
408 This captures the broadcast domain of the given network/elan.
409 When the first RT2 route is received from the dc gw,
410 it's tep ip is added to the elan to which this RT2 route belongs to.
415 augment "/elan:elan-instances/elan:elan-instance" {
416 ext:augment-identifier "evpn";
425 container elan-instances {
427 key "elan-instance-name";
428 leaf elan-instance-name {
431 //omitted other existing fields
435 type inet:ip-address;
441 container elan-interfaces {
442 list elan-interface {
446 path "/if:interfaces/if:interface/if:name";
449 leaf elan-instance-name {
453 list static-mac-entries {
456 type yang:phys-address;
458 leaf prefix {//new field
460 type inet:ip-address;
466 grouping forwarding-entries {
470 type yang:phys-address;
474 path "/if:interfaces/if:interface/if:name";
477 leaf controllerLearnedForwardingEntryTimestamp {
480 leaf isStaticAddress {
483 leaf prefix {//new field
485 type inet:ip-address;
490 Solution considerations
491 -----------------------
493 Proposed change in Openstack Neutron BGPVPN Driver
494 +++++++++++++++++++++++++++++++++++++++++++++++++++
495 The Openstack Neutron BGPVPN’s ODL driver in Newton release is changed (mitaka release), so that
496 it is able to relay the configured L2 BGPVPNs, to the OpenDaylight Controller.
498 The Newton changes for the BGPVPN Driver has merged and is here:
499 https://review.openstack.org/#/c/370547/
501 Proposed change in BGP Quagga Stack
502 ++++++++++++++++++++++++++++++++++++
503 The BGP Quagga Stack is a component that interfaces with ODL Controller to enable ODL Controller itself
504 to become a BGP Peer. This BGP Quagga Stack need to be enhanced so that it is able to embrace EVPN
505 with Route Type 5 on the following two interfaces:
507 * Thrift Interface where ODL pushes routes to BGP Quagga Stack
508 * Route exchanges from BGP Quagga Stack to other BGP Neighbors (including DC-GW).
510 Proposed change in OpenDaylight-specific features
511 +++++++++++++++++++++++++++++++++++++++++++++++++
513 The following components within OpenDaylight Controller needs to be enhanced:
516 * VPN Engine (VPN Manager)
523 This feature support all the following Reboot Scenarios for EVPN:
525 * Entire Cluster Reboot
527 * Candidate PL reboot
528 * OVS Datapath reboots
529 * Multiple PL reboots
530 * Multiple Cluster reboots
531 * Multiple reboots of the same OVS Datapath.
532 * Openstack Controller reboots
537 The following parameters have been initially made available as configurable for EVPN. These
538 configurations can be made via the RESTful interface:
540 **1.Multi-homing-mode** – For multi-homing use cases where redundant DCGWs are used ODL can be configured with ‘none’, ‘all-active’ or ‘single-active’ multi-homing mode. Default will be ‘none’.
542 **2.IRB-mode** – Depending upon the support on DCGW, ODL can be configured with either ‘Symmetric’ or ‘Asymmetric’ IRB mode. Default is ‘Symmetric’.
544 There is another important parameter though it won’t be configurable:
546 **MAC Address Prefix for EVPN** – This MAC Address prefix represents the MAC Address prefix that will be hardcoded and that MACAddress will be used as the gateway mac address if it is not supplied from Openstack. This will usually be the case when networks are associated to an L3VPN with no gateway port yet configured in Openstack for such networks.
549 Clustering considerations
550 -------------------------
551 The feature should operate in ODL Clustered environment reliably.
553 Other Infra considerations
554 --------------------------
557 Security considerations
558 -----------------------
561 Scale and Performance Impact
562 ----------------------------
563 Not covered by this Design Document.
571 Alternatives considered and why they were not selected.
578 This feature can be used by installing odl-netvirt-openstack.
579 This feature doesn't add any new karaf feature.
583 A new rpc is added to create and delete evpn:
590 'export-RT': ['50:2'],
591 'route-distinguisher': ['50:2'],
592 'import-RT': ['50:2'],
593 'id': '4ae8cd92-48ca-49b5-94e1-b2921a260007',
595 'tenant-id': 'a565b3ed854247f795c0840b0481c699'
598 There is no change in the REST API for associating networks to the EVPN.
600 On the Openstack-side configuration, the vni_ranges configured in Openstack Neutron ml2_conf.ini
601 should not overlap with the L3VNI provided in the ODL RESTful API.
602 In an inter-DC case, where both the DCs are managed by two different Openstack Controller
603 Instances, the workflow will be to do the following:
605 1. Configure the DC-GW2 facing OSC2 (Openstack) and DC-GW1 facing OSC1 with the same BGP configuration parameters.
606 2. On first Openstack Controller (OSC1) create an L3VPN1 with RD1 and L3VNI1
607 3. On first Openstack Controller (OSC1) create an EVPN1 with RD2 and L2VNI1
608 4. Create a network Net1 and Associate that Network Net1 to L3VPN1
609 5. Create a network Net1 and Associate that Network Net1 to EVPN1
610 6. On second Openstack Controller (OSC2) create an L3VPN2 with RD1 with L3VNI1
611 7. On second Openstack Controller (OSC2) create an EVPN2 with RD2 with L2VNI1
612 8. Create a network Net2 on OSC2 with same cidr as the first one with a different allocation pool and associate that Network Net2 to L3VPN2.
613 9. Associate that Network Net2 to EVPN2.
614 10. Spin-off VM1 on Net1 in OSC1.
615 11. Spin-off VM2 on Net2 in OSC2.
616 12. Now VM1 and VM2 should be able to communicate.
626 Vyshakh Krishnan C H <vyshakh.krishnan.c.h@ericsson.com>
628 Yugandhar Reddy Kaku <yugandhar.reddy.kaku@ericsson.com>
630 Riyazahmed D Talikoti <riyazahmed.d.talikoti@ericsson.com>
633 K.V Suneelu Verma <k.v.suneelu.verma@ericsson.com>
637 Trello card details https://trello.com/c/PysPZscm/150-evpn-evpn-rt2.
641 Requires a DC-GW that is supporting EVPN RT2 on BGP Control plane.
645 Capture details of testing that will need to be added.
649 Appropriate UTs will be added for the new code coming in once framework is in place.
653 There won't be any Integration tests provided for this feature.
657 CSIT will be enhanced to cover this feature by providing new CSIT tests.
661 This will require changes to User Guide and Developer Guide.
665 [1] `EVPN_RT5 <https://tools.ietf.org/html/draft-ietf-bess-evpn-prefix-advertisement-03>`_
667 [2] `Network Virtualization using EVPN <https://www.ietf.org/id/draft-ietf-bess-evpn-overlay-07.txt>`_
669 [3] `Integrated Routing and Bridging in EVPN <https://tools.ietf.org/html/draft-ietf-bess-evpn-inter-subnet-forwarding-04>`_
671 [4] `VXLAN DCI using EVPN <https://tools.ietf.org/html/draft-boutros-bess-vxlan-evpn-02>`_
673 [5] `BGP MPLS-Based Ethernet VPN <https://tools.ietf.org/html/rfc7432>`_
675 [6] `Trello card details <https://trello.com/c/PysPZscm/150-evpn-evpn-rt2>`_