| CARVIEW |
Select Language
HTTP/2 200
date: Sat, 11 Oct 2025 09:37:47 GMT
content-type: text/plain; charset=utf-8
content-length: 21622
cf-ray: 98cd6178ddc18087-BLR
last-modified: Fri, 30 Mar 2018 01:00:00 GMT
etag: "12bde-56896c27801a7-gzip"
vary: Accept-Encoding
content-encoding: gzip
strict-transport-security: max-age=31536000; includeSubDomains
x-frame-options: SAMEORIGIN
x-xss-protection: 1; mode=block
x-content-type-options: nosniff
cf-cache-status: HIT
expires: Sat, 11 Oct 2025 09:57:47 GMT
cache-control: public, max-age=1200
accept-ranges: bytes
set-cookie: __cf_bm=QWGsejzRKrYTypyLFUSTOX0f.vOXxOl2CF._wdvHnBE-1760175467-1.0.1.1-yuePaZEq3aVh_7ZmaVSy4kVawfcikuWS9ksY2CwJB_X9.LUsO0cYOxQTBCz.he.H2GpNrce5kqfLh3k4iFdTDlB8pd70Iy5Cq5g1zRoxoeo; path=/; expires=Sat, 11-Oct-25 10:07:47 GMT; domain=.rfc-editor.org; HttpOnly; Secure; SameSite=None
server: cloudflare
alt-svc: h3=":443"; ma=86400
Internet Engineering Task Force (IETF) A. Sajassi, Ed.
Request for Comments: 8365 Cisco
Category: Standards Track J. Drake, Ed.
ISSN: 2070-1721 Juniper
N. Bitar
Nokia
R. Shekhar
Juniper
J. Uttaro
AT&T
W. Henderickx
Nokia
March 2018
A Network Virtualization Overlay Solution Using Ethernet VPN (EVPN)
Abstract
This document specifies how Ethernet VPN (EVPN) can be used as a
Network Virtualization Overlay (NVO) solution and explores the
various tunnel encapsulation options over IP and their impact on the
EVPN control plane and procedures. In particular, the following
encapsulation options are analyzed: Virtual Extensible LAN (VXLAN),
Network Virtualization using Generic Routing Encapsulation (NVGRE),
and MPLS over GRE. This specification is also applicable to Generic
Network Virtualization Encapsulation (GENEVE); however, some
incremental work is required, which will be covered in a separate
document. This document also specifies new multihoming procedures
for split-horizon filtering and mass withdrawal. It also specifies
EVPN route constructions for VXLAN/NVGRE encapsulations and
Autonomous System Border Router (ASBR) procedures for multihoming of
Network Virtualization Edge (NVE) devices.
Status of This Memo
This is an Internet Standards Track document.
This document is a product of the Internet Engineering Task Force
(IETF). It represents the consensus of the IETF community. It has
received public review and has been approved for publication by the
Internet Engineering Steering Group (IESG). Further information on
Internet Standards is available in Section 2 of RFC 7841.
Information about the current status of this document, any errata,
and how to provide feedback on it may be obtained at
https://www.rfc-editor.org/info/rfc8365.
Sajassi, et al. Standards Track [Page 1]
RFC 8365 Network Virtualization Overlay Solution March 2018
Copyright Notice
Copyright (c) 2018 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents
(https://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License.
Sajassi, et al. Standards Track [Page 2]
RFC 8365 Network Virtualization Overlay Solution March 2018
Table of Contents
1. Introduction ....................................................4
2. Requirements Notation and Conventions ...........................5
3. Terminology .....................................................5
4. EVPN Features ...................................................7
5. Encapsulation Options for EVPN Overlays .........................8
5.1. VXLAN/NVGRE Encapsulation ..................................8
5.1.1. Virtual Identifiers Scope ...........................9
5.1.2. Virtual Identifiers to EVI Mapping .................11
5.1.3. Constructing EVPN BGP Routes .......................13
5.2. MPLS over GRE .............................................15
6. EVPN with Multiple Data-Plane Encapsulations ...................15
7. Single-Homing NVEs - NVE Residing in Hypervisor ................16
7.1. Impact on EVPN BGP Routes & Attributes for VXLAN/NVGRE ....16
7.2. Impact on EVPN Procedures for VXLAN/NVGRE Encapsulations ..17
8. Multihoming NVEs - NVE Residing in ToR Switch ..................18
8.1. EVPN Multihoming Features .................................18
8.1.1. Multihomed ES Auto-Discovery .......................18
8.1.2. Fast Convergence and Mass Withdrawal ...............18
8.1.3. Split-Horizon ......................................19
8.1.4. Aliasing and Backup Path ...........................19
8.1.5. DF Election ........................................20
8.2. Impact on EVPN BGP Routes and Attributes ..................20
8.3. Impact on EVPN Procedures .................................20
8.3.1. Split Horizon ......................................21
8.3.2. Aliasing and Backup Path ...........................22
8.3.3. Unknown Unicast Traffic Designation ................22
9. Support for Multicast ..........................................23
10. Data-Center Interconnections (DCIs) ...........................24
10.1. DCI Using GWs ............................................24
10.2. DCI Using ASBRs ..........................................24
10.2.1. ASBR Functionality with Single-Homing NVEs ........25
10.2.2. ASBR Functionality with Multihoming NVEs ..........26
11. Security Considerations .......................................28
12. IANA Considerations ...........................................29
13. References ....................................................29
13.1. Normative References .....................................29
13.2. Informative References ...................................30
Acknowledgements ..................................................32
Contributors ......................................................32
Authors' Addresses ................................................33
Sajassi, et al. Standards Track [Page 3]
RFC 8365 Network Virtualization Overlay Solution March 2018
1. Introduction
This document specifies how Ethernet VPN (EVPN) [RFC7432] can be used
as a Network Virtualization Overlay (NVO) solution and explores the
various tunnel encapsulation options over IP and their impact on the
EVPN control plane and procedures. In particular, the following
encapsulation options are analyzed: Virtual Extensible LAN (VXLAN)
[RFC7348], Network Virtualization using Generic Routing Encapsulation
(NVGRE) [RFC7637], and MPLS over Generic Routing Encapsulation (GRE)
[RFC4023]. This specification is also applicable to Generic Network
Virtualization Encapsulation (GENEVE) [GENEVE]; however, some
incremental work is required, which will be covered in a separate
document [EVPN-GENEVE]. This document also specifies new multihoming
procedures for split-horizon filtering and mass withdrawal. It also
specifies EVPN route constructions for VXLAN/NVGRE encapsulations and
Autonomous System Border Router (ASBR) procedures for multihoming of
Network Virtualization Edge (NVE) devices.
In the context of this document, an NVO is a solution to address the
requirements of a multi-tenant data center, especially one with
virtualized hosts, e.g., Virtual Machines (VMs) or virtual workloads.
The key requirements of such a solution, as described in [RFC7364],
are the following:
- Isolation of network traffic per tenant
- Support for a large number of tenants (tens or hundreds of
thousands)
- Extension of Layer 2 (L2) connectivity among different VMs
belonging to a given tenant segment (subnet) across different
Points of Delivery (PoDs) within a data center or between
different data centers
- Allowing a given VM to move between different physical points of
attachment within a given L2 segment
The underlay network for NVO solutions is assumed to provide IP
connectivity between NVO endpoints.
Sajassi, et al. Standards Track [Page 4]
RFC 8365 Network Virtualization Overlay Solution March 2018
This document describes how EVPN can be used as an NVO solution and
explores applicability of EVPN functions and procedures. In
particular, it describes the various tunnel encapsulation options for
EVPN over IP and their impact on the EVPN control plane as well as
procedures for two main scenarios:
(a) single-homing NVEs - when an NVE resides in the hypervisor, and
(b) multihoming NVEs - when an NVE resides in a Top-of-Rack (ToR)
device.
The possible encapsulation options for EVPN overlays that are
analyzed in this document are:
- VXLAN and NVGRE
- MPLS over GRE
Before getting into the description of the different encapsulation
options for EVPN over IP, it is important to highlight the EVPN
solution's main features, how those features are currently supported,
and any impact that the encapsulation has on those features.
2. Requirements Notation and Conventions
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
"OPTIONAL" in this document are to be interpreted as described in
BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all
capitals, as shown here.
3. Terminology
Most of the terminology used in this documents comes from [RFC7432]
and [RFC7365].
VXLAN: Virtual Extensible LAN
GRE: Generic Routing Encapsulation
NVGRE: Network Virtualization using Generic Routing Encapsulation
GENEVE: Generic Network Virtualization Encapsulation
PoD: Point of Delivery
NV: Network Virtualization
Sajassi, et al. Standards Track [Page 5]
RFC 8365 Network Virtualization Overlay Solution March 2018
NVO: Network Virtualization Overlay
NVE: Network Virtualization Edge
VNI: VXLAN Network Identifier
VSID: Virtual Subnet Identifier (for NVGRE)
I-SID: Service Instance Identifier
EVPN: Ethernet VPN
EVI: EVPN Instance. An EVPN instance spanning the Provider Edge
(PE) devices participating in that EVPN
MAC-VRF: A Virtual Routing and Forwarding table for Media Access
Control (MAC) addresses on a PE
IP-VRF: A Virtual Routing and Forwarding table for Internet Protocol
(IP) addresses on a PE
ES: Ethernet Segment. When a customer site (device or network) is
connected to one or more PEs via a set of Ethernet links, then
that set of links is referred to as an 'Ethernet segment'.
Ethernet Segment Identifier (ESI): A unique non-zero identifier that
identifies an Ethernet segment is called an 'Ethernet Segment
Identifier'.
Ethernet Tag: An Ethernet tag identifies a particular broadcast
domain, e.g., a VLAN. An EVPN instance consists of one or more
broadcast domains.
PE: Provider Edge
Single-Active Redundancy Mode: When only a single PE, among all the
PEs attached to an ES, is allowed to forward traffic to/from that
ES for a given VLAN, then the Ethernet segment is defined to be
operating in Single-Active redundancy mode.
All-Active Redundancy Mode: When all PEs attached to an Ethernet
segment are allowed to forward known unicast traffic to/from that
ES for a given VLAN, then the ES is defined to be operating in
All-Active redundancy mode.
PIM-SM: Protocol Independent Multicast - Sparse-Mode
Sajassi, et al. Standards Track [Page 6]
RFC 8365 Network Virtualization Overlay Solution March 2018
PIM-SSM: Protocol Independent Multicast - Source-Specific Multicast
BIDIR-PIM: Bidirectional PIM
4. EVPN Features
EVPN [RFC7432] was originally designed to support the requirements
detailed in [RFC7209] and therefore has the following attributes
which directly address control-plane scaling and ease of deployment
issues.
1. Control-plane information is distributed with BGP and broadcast
and multicast traffic is sent using a shared multicast tree or
with ingress replication.
2. Control-plane learning is used for MAC (and IP) addresses
instead of data-plane learning. The latter requires the
flooding of unknown unicast and Address Resolution Protocol
(ARP) frames; whereas, the former does not require any flooding.
3. Route Reflector (RR) is used to reduce a full mesh of BGP
sessions among PE devices to a single BGP session between a PE
and the RR. Furthermore, RR hierarchy can be leveraged to scale
the number of BGP routes on the RR.
4. Auto-discovery via BGP is used to discover PE devices
participating in a given VPN, PE devices participating in a
given redundancy group, tunnel encapsulation types, multicast
tunnel types, multicast members, etc.
5. All-Active multihoming is used. This allows a given Customer
Edge (CE) device to have multiple links to multiple PEs, and
traffic to/from that CE fully utilizes all of these links.
6. When a link between a CE and a PE fails, the PEs for that EVI
are notified of the failure via the withdrawal of a single EVPN
route. This allows those PEs to remove the withdrawing PE as a
next hop for every MAC address associated with the failed link.
This is termed "mass withdrawal".
7. BGP route filtering and constrained route distribution are
leveraged to ensure that the control-plane traffic for a given
EVI is only distributed to the PEs in that EVI.
Sajassi, et al. Standards Track [Page 7]
RFC 8365 Network Virtualization Overlay Solution March 2018
8. When an IEEE 802.1Q [IEEE.802.1Q] interface is used between a CE
and a PE, each of the VLAN IDs (VIDs) on that interface can be
mapped onto a bridge table (for up to 4094 such bridge tables).
All these bridge tables may be mapped onto a single MAC-VRF (in
case of VLAN-aware bundle service).
9. VM Mobility mechanisms ensure that all PEs in a given EVI know
the ES with which a given VM, as identified by its MAC and IP
addresses, is currently associated.
10. RTs are used to allow the operator (or customer) to define a
spectrum of logical network topologies including mesh, hub and
spoke, and extranets (e.g., a VPN whose sites are owned by
different enterprises), without the need for proprietary
software or the aid of other virtual or physical devices.
Because the design goal for NVO is millions of instances per common
physical infrastructure, the scaling properties of the control plane
for NVO are extremely important. EVPN and the extensions described
herein, are designed with this level of scalability in mind.
5. Encapsulation Options for EVPN Overlays
5.1. VXLAN/NVGRE Encapsulation
Both VXLAN and NVGRE are examples of technologies that provide a data
plane encapsulation which is used to transport a packet over the
common physical IP infrastructure between Network Virtualization
Edges (NVEs) - e.g., VXLAN Tunnel End Points (VTEPs) in VXLAN
network. Both of these technologies include the identifier of the
specific NVO instance, VNI in VXLAN and VSID in NVGRE, in each
packet. In the remainder of this document we use VNI as the
representation for NVO instance with the understanding that VSID can
equally be used if the encapsulation is NVGRE unless it is stated
otherwise.
Note that a PE is equivalent to an NVE/VTEP.
VXLAN encapsulation is based on UDP, with an 8-byte header following
the UDP header. VXLAN provides a 24-bit VNI, which typically
provides a one-to-one mapping to the tenant VID, as described in
[RFC7348]. In this scenario, the ingress VTEP does not include an
inner VLAN tag on the encapsulated frame, and the egress VTEP
discards the frames with an inner VLAN tag. This mode of operation
in [RFC7348] maps to VLAN-Based Service in [RFC7432], where a tenant
VID gets mapped to an EVI.
Sajassi, et al. Standards Track [Page 8]
RFC 8365 Network Virtualization Overlay Solution March 2018
VXLAN also provides an option of including an inner VLAN tag in the
encapsulated frame, if explicitly configured at the VTEP. This mode
of operation can map to VLAN Bundle Service in [RFC7432] because all
the tenant's tagged frames map to a single bridge table / MAC-VRF,
and the inner VLAN tag is not used for lookup by the disposition PE
when performing VXLAN decapsulation as described in Section 6 of
[RFC7348].
[RFC7637] encapsulation is based on GRE encapsulation, and it
mandates the inclusion of the optional GRE Key field, which carries
the VSID. There is a one-to-one mapping between the VSID and the
tenant VID, as described in [RFC7637]. The inclusion of an inner
VLAN tag is prohibited. This mode of operation in [RFC7637] maps to
VLAN Based Service in [RFC7432].
As described in the next section, there is no change to the encoding
of EVPN routes to support VXLAN or NVGRE encapsulation, except for
the use of the BGP Encapsulation Extended Community to indicate the
encapsulation type (e.g., VXLAN or NVGRE). However, there is
potential impact to the EVPN procedures depending on where the NVE is
located (i.e., in hypervisor or ToR) and whether multihoming
capabilities are required.
5.1.1. Virtual Identifiers Scope
Although VNIs are defined as 24-bit globally unique values, there are
scenarios in which it is desirable to use a locally significant value
for the VNI, especially in the context of a data-center interconnect.
5.1.1.1. Data-Center Interconnect with Gateway
In the case where NVEs in different data centers need to be
interconnected, and the NVEs need to use VNIs as globally unique
identifiers within a data center, then a Gateway (GW) needs to be
employed at the edge of the data-center network (DCN). This is
because the Gateway will provide the functionality of translating the
VNI when crossing network boundaries, which may align with operator
span-of-control boundaries. As an example, consider the network of
Figure 1. Assume there are three network operators: one for each of
the DC1, DC2, and WAN networks. The Gateways at the edge of the data
centers are responsible for translating the VNIs between the values
used in each of the DCNs and the values used in the WAN.
Sajassi, et al. Standards Track [Page 9]
RFC 8365 Network Virtualization Overlay Solution March 2018
+--------------+
| |
+---------+ | WAN | +---------+
+----+ | +---+ +----+ +----+ +---+ | +----+
|NVE1|--| | | |WAN | |WAN | | | |--|NVE3|
+----+ |IP |GW |--|Edge| |Edge|--|GW | IP | +----+
+----+ |Fabric +---+ +----+ +----+ +---+ Fabric | +----+
|NVE2|--| | | | | |--|NVE4|
+----+ +---------+ +--------------+ +---------+ +----+
|<------ DC 1 ------> <------ DC2 ------>|
Figure 1: Data-Center Interconnect with Gateway
5.1.1.2. Data-Center Interconnect without Gateway
In the case where NVEs in different data centers need to be
interconnected, and the NVEs need to use locally assigned VNIs (e.g.,
similar to MPLS labels), there may be no need to employ Gateways at
the edge of the DCN. More specifically, the VNI value that is used
by the transmitting NVE is allocated by the NVE that is receiving the
traffic (in other words, this is similar to a "downstream-assigned"
MPLS label). This allows the VNI space to be decoupled between
different DCNs without the need for a dedicated Gateway at the edge
of the data centers. This topic is covered in Section 10.2.
+--------------+
| |
+---------+ | WAN | +---------+
+----+ | | +----+ +----+ | | +----+
|NVE1|--| | |ASBR| |ASBR| | |--|NVE3|
+----+ |IP Fabric|---| | | |--|IP Fabric| +----+
+----+ | | +----+ +----+ | | +----+
|NVE2|--| | | | | |--|NVE4|
+----+ +---------+ +--------------+ +---------+ +----+
|<------ DC 1 -----> <---- DC2 ------>|
Figure 2: Data-Center Interconnect with ASBR
Sajassi, et al. Standards Track [Page 10]
RFC 8365 Network Virtualization Overlay Solution March 2018
5.1.2. Virtual Identifiers to EVI Mapping
Just like in [RFC7432], where two options existed for mapping
broadcast domains (represented by VLAN IDs) to an EVI, when the EVPN
control plane is used in conjunction with VXLAN (or NVGRE
encapsulation), there are also two options for mapping broadcast
domains represented by VXLAN VNIs (or NVGRE VSIDs) to an EVI:
Option 1: A Single Broadcast Domain per EVI
In this option, a single Ethernet broadcast domain (e.g., subnet)
represented by a VNI is mapped to a unique EVI. This corresponds to
the VLAN-Based Service in [RFC7432], where a tenant-facing interface,
logical interface (e.g., represented by a VID), or physical interface
gets mapped to an EVI. As such, a BGP Route Distinguisher (RD) and
Route Target (RT) are needed per VNI on every NVE. The advantage of
this model is that it allows the BGP RT constraint mechanisms to be
used in order to limit the propagation and import of routes to only
the NVEs that are interested in a given VNI. The disadvantage of
this model may be the provisioning overhead if the RD and RT are not
derived automatically from the VNI.
In this option, the MAC-VRF table is identified by the RT in the
control plane and by the VNI in the data plane. In this option, the
specific MAC-VRF table corresponds to only a single bridge table.
Option 2: Multiple Broadcast Domains per EVI
In this option, multiple subnets, each represented by a unique VNI,
are mapped to a single EVI. For example, if a tenant has multiple
segments/subnets each represented by a VNI, then all the VNIs for
that tenant are mapped to a single EVI; for example, the EVI in this
case represents the tenant and not a subnet. This corresponds to the
VLAN-aware bundle service in [RFC7432]. The advantage of this model
is that it doesn't require the provisioning of an RD/RT per VNI.
However, this is a moot point when compared to Option 1 where auto-
derivation is used. The disadvantage of this model is that routes
would be imported by NVEs that may not be interested in a given VNI.
In this option, the MAC-VRF table is identified by the RT in the
control plane; a specific bridge table for that MAC-VRF is identified
by the in the control plane. In this option,
the VNI in the data plane is sufficient to identify a specific bridge
table.
Sajassi, et al. Standards Track [Page 11]
RFC 8365 Network Virtualization Overlay Solution March 2018
5.1.2.1. Auto-Derivation of RT
In order to simplify configuration, when the option of a single VNI
per EVI is used, the RT used for EVPN can be auto-derived. RD can be
auto-generated as described in [RFC7432], and RT can be auto-derived
as described next.
Since a Gateway PE as depicted in Figure 1 participates in both the
DCN and WAN BGP sessions, it is important that, when RT values are
auto-derived from VNIs, there be no conflict in RT spaces between
DCNs and WANs, assuming that both are operating within the same
Autonomous System (AS). Also, there can be scenarios where both
VXLAN and NVGRE encapsulations may be needed within the same DCN, and
their corresponding VNIs are administered independently, which means
VNI spaces can overlap. In order to avoid conflict in RT spaces, the
6-byte RT values with 2-octet AS number for DCNs can be auto-derived
as follow:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Global Administrator | Local Administrator |
+-----------------------------------------------+---------------+
| Local Administrator (Cont.) |
+-------------------------------+
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Global Administrator |A| TYPE| D-ID | Service ID |
+-----------------------------------------------+---------------+
| Service ID (Cont.) |
+-------------------------------+
The 6-octet RT field consists of two sub-fields:
- Global Administrator sub-field: 2 octets. This sub-field contains
an AS number assigned by IANA for VLAN-Aware Bundle Service.
All fields of this route are set as described in Section 5.1.3. The
originating router's IP address field is set to the NVE's IP address.
This route is tagged with the PMSI Tunnel attribute, which is used to
encode the type of multicast tunnel to be used as well as the
multicast tunnel identifier. The tunnel encapsulation is encoded by
adding the BGP Encapsulation Extended Community as per Section 5.1.1.
For example, the PMSI Tunnel attribute may indicate the multicast
tunnel is of type Protocol Independent Multicast - Sparse-Mode (PIM-
SM); whereas, the BGP Encapsulation Extended Community may indicate
the encapsulation for that tunnel is of type VXLAN. The following
tunnel types as defined in [RFC6514] can be used in the PMSI Tunnel
attribute for VXLAN/NVGRE:
+ 3 - PIM-SSM Tree
+ 4 - PIM-SM Tree
+ 5 - BIDIR-PIM Tree
+ 6 - Ingress Replication
In case of VXLAN and NVGRE encapsulations with locally assigned VNIs,
just as in [RFC7432], each PE MUST advertise an IMET route to other
PEs in an EVPN instance for the multicast tunnel type that it uses
(i.e., ingress replication, PIM-SM, PIM-SSM, or BIDIR-PIM tunnel).
However, for globally assigned VNIs, each PE MUST advertise an IMET
route to other PEs in an EVPN instance for ingress replication or a
PIM-SSM tunnel, and they MAY advertise an IMET route for a PIM-SM or
BIDIR-PIM tunnel. In case of a PIM-SM or BIDIR-PIM tunnel, no
information in the IMET route is needed by the PE to set up these
tunnels.
In the scenario where the multicast tunnel is a tree, both the
Inclusive as well as the Aggregate Inclusive variants may be used.
In the former case, a multicast tree is dedicated to a VNI. Whereas,
in the latter, a multicast tree is shared among multiple VNIs. For
VNI-Based Service, the Aggregate Inclusive mode is accomplished by
having the NVEs advertise multiple IMET routes with different RTs
(one per VNI) but with the same tunnel identifier encoded in the PMSI
Tunnel attribute. For VNI-Aware Bundle Service, the Aggregate
Inclusive mode is accomplished by having the NVEs advertise multiple
IMET routes with different VNIs encoded in the Ethernet Tag field,
but with the same tunnel identifier encoded in the PMSI Tunnel
attribute.
Sajassi, et al. Standards Track [Page 23]
RFC 8365 Network Virtualization Overlay Solution March 2018
10. Data-Center Interconnections (DCIs)
For DCIs, the following two main scenarios are considered when
connecting data centers running evpn-overlay (as described here) over
an MPLS/IP core network:
- Scenario 1: DCI using GWs
- Scenario 2: DCI using ASBRs
The following two subsections describe the operations for each of
these scenarios.
10.1. DCI Using GWs
This is the typical scenario for interconnecting data centers over
WAN. In this scenario, EVPN routes are terminated and processed in
each GW and MAC/IP route are always re-advertised from DC to WAN but
from WAN to DC, they are not re-advertised if unknown MAC addresses
(and default IP address) are utilized in the NVEs. In this scenario,
each GW maintains a MAC-VRF (and/or IP-VRF) for each EVI. The main
advantage of this approach is that NVEs do not need to maintain MAC
and IP addresses from any remote data centers when default IP routes
and unknown MAC routes are used; that is, they only need to maintain
routes that are local to their own DC. When default IP routes and
unknown MAC routes are used, any unknown IP and MAC packets from NVEs
are forwarded to the GWs where all the VPN MAC and IP routes are
maintained. This approach reduces the size of MAC-VRF and IP-VRF
significantly at NVEs. Furthermore, it results in a faster
convergence time upon a link or NVE failure in a multihomed network
or device redundancy scenario, because the failure-related BGP routes
(such as mass withdrawal message) do not need to get propagated all
the way to the remote NVEs in the remote DCs. This approach is
described in detail in Section 3.4 of [DCI-EVPN-OVERLAY].
10.2. DCI Using ASBRs
This approach can be considered as the opposite of the first
approach. It favors simplification at DCI devices over NVEs such
that larger MAC-VRF (and IP-VRF) tables need to be maintained on
NVEs; whereas DCI devices don't need to maintain any MAC (and IP)
forwarding tables. Furthermore, DCI devices do not need to terminate
and process routes related to multihoming but rather to relay these
messages for the establishment of an end-to-end Label Switched Path
(LSP). In other words, DCI devices in this approach operate similar
to ASBRs for inter-AS Option B (see Section 10 of [RFC4364]). This
requires locally assigned VNIs to be used just like downstream-
assigned MPLS VPN labels where, for all practical purposes, the VNIs
Sajassi, et al. Standards Track [Page 24]
RFC 8365 Network Virtualization Overlay Solution March 2018
function like 24-bit VPN labels. This approach is equally applicable
to data centers (or Carrier Ethernet networks) with MPLS
encapsulation.
In inter-AS Option B, when ASBR receives an EVPN route from its DC
over internal BGP (iBGP) and re-advertises it to other ASBRs, it
re-advertises the EVPN route by re-writing the BGP next hops to
itself, thus losing the identity of the PE that originated the
advertisement. This rewrite of BGP next hop impacts the EVPN mass
withdrawal route (Ethernet A-D per ES) and its procedure adversely.
However, it does not impact the EVPN Aliasing mechanism/procedure
because when the Aliasing routes (Ethernet A-D per EVI) are
advertised, the receiving PE first resolves a MAC address for a given
EVI into its corresponding , and, subsequently, it resolves
the into multiple paths (and their associated next hops)
via which the is reachable. Since Aliasing and MAC routes
are both advertised on a per-EVI-basis and they use the same RD and
RT (per EVI), the receiving PE can associate them together on a
per-BGP-path basis (e.g., per originating PE). Thus, it can perform
recursive route resolution, e.g., a MAC is reachable via an
which in turn, is reachable via a set of BGP paths; thus, the MAC is
reachable via the set of BGP paths. Due to the per-EVI basis, the
association of MAC routes and the corresponding Aliasing route is
fixed and determined by the same RD and RT; there is no ambiguity
when the BGP next hop for these routes is rewritten as these routes
pass through ASBRs. That is, the receiving PE may receive multiple
Aliasing routes for the same EVI from a single next hop (a single
ASBR), and it can still create multiple paths toward that .
However, when the BGP next-hop address corresponding to the
originating PE is rewritten, the association between the mass
withdrawal route (Ethernet A-D per ES) and its corresponding MAC
routes cannot be made based on their RDs and RTs because the RD for
the mass Withdrawal route is different than the one for the MAC
routes. Therefore, the functionality needed at the ASBRs and the
receiving PEs depends on whether the Mass Withdrawal route is
originated and whether there is a need to handle route resolution
ambiguity for this route. The following two subsections describe the
functionality needed by the ASBRs and the receiving PEs depending on
whether the NVEs reside in a hypervisors or in ToR switches.
10.2.1. ASBR Functionality with Single-Homing NVEs
When NVEs reside in hypervisors as described in Section 7.1, there is
no multihoming; thus, there is no need for the originating NVE to
send Ethernet A-D per ES or Ethernet A-D per EVI routes. However, as
noted in Section 7, in order to enable a single-homing ingress NVE to
take advantage of fast convergence, Aliasing, and Backup Path when
Sajassi, et al. Standards Track [Page 25]
RFC 8365 Network Virtualization Overlay Solution March 2018
interacting with multihoming egress NVEs attached to a given ES, the
single-homing NVE should be able to receive and process Ethernet A-D
per ES and Ethernet A-D per EVI routes. The handling of these routes
is described in the next section.
10.2.2. ASBR Functionality with Multihoming NVEs
When NVEs reside in ToR switches and operate in multihoming
redundancy mode, there is a need, as described in Section 8, for the
originating multihoming NVE to send Ethernet A-D per ES route(s)
(used for mass withdrawal) and Ethernet A-D per EVI routes (used for
Aliasing). As described above, the rewrite of BGP next hop by ASBRs
creates ambiguities when Ethernet A-D per ES routes are received by
the remote NVE in a different ASBR because the receiving NVE cannot
associate that route with the MAC/IP routes of that ES advertised by
the same originating NVE. This ambiguity inhibits the function of
mass withdrawal per ES by the receiving NVE in a different AS.
As an example, consider a scenario where a CE is multihomed to PE1
and PE2, where these PEs are connected via ASBR1 and then ASBR2 to
the remote PE3. Furthermore, consider that PE1 receives M1 from CE1
but not PE2. Therefore, PE1 advertises Ethernet A-D per ES1,
Ethernet A-D per EVI1, and M1; whereas, PE2 only advertises Ethernet
A-D per ES1 and Ethernet A-D per EVI1. ASBR1 receives all these five
advertisements and passes them to ASBR2 (with itself as the BGP next
hop). ASBR2, in turn, passes them to the remote PE3, with itself as
the BGP next hop. PE3 receives these five routes where all of them
have the same BGP next hop (i.e., ASBR2). Furthermore, the two
Ethernet A-D per ES routes received by PE3 have the same information,
i.e., same ESI and the same BGP next hop. Although both of these
routes are maintained by the BGP process in PE3 (because they have
different RDs and, thus, are treated as different BGP routes),
information from only one of them is used in the L2 routing table (L2
RIB).
PE1
/ \
CE ASBR1---ASBR2---PE3
\ /
PE2
Figure 3: Inter-AS Option B
Now, when the AC between the PE2 and the CE fails and PE2 sends
Network Layer Reachability Information (NLRI) withdrawal for Ethernet
A-D per ES route, and this withdrawal gets propagated and received by
the PE3, the BGP process in PE3 removes the corresponding BGP route;
however, it doesn't remove the associated information (namely ESI and
Sajassi, et al. Standards Track [Page 26]
RFC 8365 Network Virtualization Overlay Solution March 2018
BGP next hop) from the L2 routing table (L2 RIB) because it still has
the other Ethernet A-D per ES route (originated from PE1) with the
same information. That is why the mass withdrawal mechanism does not
work when doing DCI with inter-AS Option B. However, as described
previously, the Aliasing function works and so does "mass withdrawal
per EVI" (which is associated with withdrawing the EVPN route
associated with Aliasing, i.e., Ethernet A-D per EVI route).
In the above example, the PE3 receives two Aliasing routes with the
same BGP next hop (ASBR2) but different RDs. One of the Aliasing
route has the same RD as the advertised MAC route (M1). PE3 follows
the route resolution procedure specified in [RFC7432] upon receiving
the two Aliasing routes; that is, it resolves M1 to , and,
subsequently, it resolves to a BGP path list with two
paths along with the corresponding VNIs/MPLS labels (one associated
with PE1 and the other associated with PE2). It should be noted that
even though both paths are advertised by the same BGP next hop
(ASRB2), the receiving PE3 can handle them properly. Therefore, M1
is reachable via two paths. This creates two end-to-end LSPs, from
PE3 to PE1 and from PE3 to PE2, for M1 such that when PE3 wants to
forward traffic destined to M1, it can load-balance between the two
LSPs. Although route resolution for Aliasing routes with the same
BGP next hop is not explicitly mentioned in [RFC7432], this is the
expected operation; thus, it is elaborated here.
When the AC between the PE2 and the CE fails and PE2 sends NLRI
withdrawal for Ethernet A-D per EVI routes, and these withdrawals get
propagated and received by the PE3, the PE3 removes the Aliasing
route and updates the path list; that is, it removes the path
corresponding to the PE2. Therefore, all the corresponding MAC
routes for that that point to that path list will now have
the updated path list with a single path associated with PE1. This
action can be considered to be the mass withdrawal at the per-EVI
level. The mass withdrawal at the per-EVI level has a longer
convergence time than the mass withdrawal at the per-ES level;
however, it is much faster than the convergence time when the
withdrawal is done on a per-MAC basis.
If a PE becomes detached from a given ES, then, in addition to
withdrawing its previously advertised Ethernet A-D per ES routes, it
MUST also withdraw its previously advertised Ethernet A-D per EVI
routes for that ES. For a remote PE that is separated from the
withdrawing PE by one or more EVPN inter-AS Option B ASBRs, the
withdrawal of the Ethernet A-D per ES routes is not actionable.
However, a remote PE is able to correlate a previously advertised
Ethernet A-D per EVI route with any MAC/IP Advertisement routes also
advertised by the withdrawing PE for that . Hence, when
Sajassi, et al. Standards Track [Page 27]
RFC 8365 Network Virtualization Overlay Solution March 2018
it receives the withdrawal of an Ethernet A-D per EVI route, it
SHOULD remove the withdrawing PE as a next hop for all MAC addresses
associated with that .
In the previous example, when the AC between PE2 and the CE fails,
PE2 will withdraw its Ethernet A-D per ES and per EVI routes. When
PE3 receives the withdrawal of an Ethernet A-D per EVI route, it
removes PE2 as a valid next hop for all MAC addresses associated with
the corresponding . Therefore, all the MAC next hops
for that will now have a single next hop, viz. the LSP
to PE1.
In summary, it can be seen that Aliasing (and Backup Path)
functionality should work as is for inter-AS Option B without
requiring any additional functionality in ASBRs or PEs. However, the
mass withdrawal functionality falls back from per-ES mode to per-EVI
mode for inter-AS Option B. That is, PEs receiving a mass withdrawal
route from the same AS take action on Ethernet A-D per ES route;
whereas, PEs receiving mass withdrawal routes from different ASes
take action on the Ethernet A-D per EVI route.
11. Security Considerations
This document uses IP-based tunnel technologies to support data-plane
transport. Consequently, the security considerations of those tunnel
technologies apply. This document defines support for VXLAN
[RFC7348] and NVGRE encapsulations [RFC7637]. The security
considerations from those RFCs apply to the data-plane aspects of
this document.
As with [RFC5512], any modification of the information that is used
to form encapsulation headers, to choose a tunnel type, or to choose
a particular tunnel for a particular payload type may lead to user
data packets getting misrouted, misdelivered, and/or dropped.
More broadly, the security considerations for the transport of IP
reachability information using BGP are discussed in [RFC4271] and
[RFC4272] and are equally applicable for the extensions described in
this document.
Sajassi, et al. Standards Track [Page 28]
RFC 8365 Network Virtualization Overlay Solution March 2018
12. IANA Considerations
This document registers the following in the "BGP Tunnel
Encapsulation Attribute Tunnel Types" registry.
Value Name
----- ------------------------
8 VXLAN Encapsulation
9 NVGRE Encapsulation
10 MPLS Encapsulation
11 MPLS in GRE Encapsulation
12 VXLAN GPE Encapsulation
13. References
13.1. Normative References
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119,
DOI 10.17487/RFC2119, March 1997,
.
[RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
May 2017, .
[RFC7432] Sajassi, A., Ed., Aggarwal, R., Bitar, N., Isaac, A.,
Uttaro, J., Drake, J., and W. Henderickx, "BGP MPLS-Based
Ethernet VPN", RFC 7432, DOI 10.17487/RFC7432, February
2015, .
[RFC7348] Mahalingam, M., Dutt, D., Duda, K., Agarwal, P., Kreeger,
L., Sridhar, T., Bursell, M., and C. Wright, "Virtual
eXtensible Local Area Network (VXLAN): A Framework for
Overlaying Virtualized Layer 2 Networks over Layer 3
Networks", RFC 7348, DOI 10.17487/RFC7348, August 2014,
.
[RFC5512] Mohapatra, P. and E. Rosen, "The BGP Encapsulation
Subsequent Address Family Identifier (SAFI) and the BGP
Tunnel Encapsulation Attribute", RFC 5512,
DOI 10.17487/RFC5512, April 2009,
.
[RFC4023] Worster, T., Rekhter, Y., and E. Rosen, Ed.,
"Encapsulating MPLS in IP or Generic Routing Encapsulation
(GRE)", RFC 4023, DOI 10.17487/RFC4023, March 2005,
.
Sajassi, et al. Standards Track [Page 29]
RFC 8365 Network Virtualization Overlay Solution March 2018
[RFC7637] Garg, P., Ed. and Y. Wang, Ed., "NVGRE: Network
Virtualization Using Generic Routing Encapsulation",
RFC 7637, DOI 10.17487/RFC7637, September 2015,
.
13.2. Informative References
[RFC7209] Sajassi, A., Aggarwal, R., Uttaro, J., Bitar, N.,
Henderickx, W., and A. Isaac, "Requirements for Ethernet
VPN (EVPN)", RFC 7209, DOI 10.17487/RFC7209, May 2014,
.
[RFC4272] Murphy, S., "BGP Security Vulnerabilities Analysis",
RFC 4272, DOI 10.17487/RFC4272, January 2006,
.
[RFC7364] Narten, T., Ed., Gray, E., Ed., Black, D., Fang, L.,
Kreeger, L., and M. Napierala, "Problem Statement:
Overlays for Network Virtualization", RFC 7364,
DOI 10.17487/RFC7364, October 2014,
.
[RFC7365] Lasserre, M., Balus, F., Morin, T., Bitar, N., and Y.
Rekhter, "Framework for Data Center (DC) Network
Virtualization", RFC 7365, DOI 10.17487/RFC7365, October
2014, .
[RFC6514] Aggarwal, R., Rosen, E., Morin, T., and Y. Rekhter, "BGP
Encodings and Procedures for Multicast in MPLS/BGP IP
VPNs", RFC 6514, DOI 10.17487/RFC6514, February 2012,
.
[RFC4271] Rekhter, Y., Ed., Li, T., Ed., and S. Hares, Ed., "A
Border Gateway Protocol 4 (BGP-4)", RFC 4271,
DOI 10.17487/RFC4271, January 2006,
.
[RFC4364] Rosen, E. and Y. Rekhter, "BGP/MPLS IP Virtual Private
Networks (VPNs)", RFC 4364, DOI 10.17487/RFC4364, February
2006, .
[TUNNEL-ENCAP]
Rosen, E., Ed., Patel, K., and G. Velde, "The BGP Tunnel
Encapsulation Attribute", Work in Progress draft-ietf-idr-
tunnel-encaps-09, February 2018.
Sajassi, et al. Standards Track [Page 30]
RFC 8365 Network Virtualization Overlay Solution March 2018
[DCI-EVPN-OVERLAY]
Rabadan, J., Ed., Sathappan, S., Henderickx, W., Sajassi,
A., and J. Drake, "Interconnect Solution for EVPN Overlay
networks", Work in Progress, draft-ietf-bess-dci-evpn-
overlay-10, March 2018.
[EVPN-GENEVE]
Boutros, S., Sajassi, A., Drake, J., and J. Rabadan, "EVPN
control plane for Geneve", Work in Progress,
draft-boutros-bess-evpn-geneve-02, March 2018.
[VXLAN-GPE]
Maino, F., Kreeger, L., Ed., and U. Elzur, Ed., "Generic
Protocol Extension for VXLAN", Work in Progress,
draft-ietf-nvo3-vxlan-gpe-05, October 2017.
[GENEVE] Gross, J., Ed., Ganga, I., Ed., and T. Sridhar, Ed.,
"Geneve: Generic Network Virtualization Encapsulation",
Work in Progress, draft-ietf-nvo3-geneve-06, March 2018.
[IEEE.802.1Q]
IEEE, "IEEE Standard for Local and metropolitan area
networks - Bridges and Bridged Networks - Media Access
Control (MAC) Bridges and Virtual Bridged Local Area
Networks", IEEE Std 802.1Q.
Sajassi, et al. Standards Track [Page 31]
RFC 8365 Network Virtualization Overlay Solution March 2018
Acknowledgements
The authors would like to thank Aldrin Isaac, David Smith, John
Mullooly, Thomas Nadeau, Samir Thoria, and Jorge Rabadan for their
valuable comments and feedback. The authors would also like to thank
Jakob Heitz for his contribution on Section 10.2.
Contributors
S. Salam
K. Patel
D. Rao
S. Thoria
D. Cai
Cisco
Y. Rekhter
A. Issac
W. Lin
N. Sheth
Juniper
L. Yong
Huawei
Sajassi, et al. Standards Track [Page 32]
RFC 8365 Network Virtualization Overlay Solution March 2018
Authors' Addresses
Ali Sajassi (editor)
Cisco
United States of America
Email: sajassi@cisco.com
John Drake (editor)
Juniper Networks
United States of America
Email: jdrake@juniper.net
Nabil Bitar
Nokia
United States of America
Email: nabil.bitar@nokia.com
R. Shekhar
Juniper
United States of America
Email: rshekhar@juniper.net
James Uttaro
AT&T
United States of America
Email: uttaro@att.com
Wim Henderickx
Nokia
Copernicuslaan 50
2018 Antwerp
Belgium
Email: wim.henderickx@nokia.com
Sajassi, et al. Standards Track [Page 33]