Authors Note: I wrote 95% of this post just after VmWorld and just prior to the announcement of NVGRE. I was too lazy to go back and add something on this new proposed standard but as a different scheme for tunneling layer 2 over layer 3, it plays directly into the theme of this piece.
In the welcome post for The Whole Stack I wrote a bit about layering and abstractions. To that end, VXLAN was introduced with great fan fair at VmWorld. This announcement made me realize how similarly the data center network is evolving, as compared to what we have already seen in carrier IP backbones, specifically with Pseudo Wires and VPLS.
As someone who got into technology via networking, the venerable OSI model is often my guide. Of course this is a just a reference model and not a strict guide or a reflection of actual practice. The idea of “this over that networking,” and “almost anything over IP,” are marketing and technical truisms.
VXLAN Inverts the Layers
The announcement of VXLAN at VmWorld follows in this great tradition of playing fast and loose with layering. VXLAN is about encapsulating Layer 2 networks for virtual machines over Layer 3 networks. The encapsulation tunnels MAC addresses in IP/UDP with the addition of a new VXLAN Network Identifier. Interestingly, this draft punts on the complex control plane issues VXLAN brings up. I suspect this fact was not lost on the start-ups trying to build external networking controllers.
VXLAN is attempting to solve a number of challenging issues for building large scale networks that support virtualized server environments. VXLAN enables the use of IP as a transport method but still preserves the Layer 2 connectivity model required for VMs for service like vMotion and DRS. VXLAN also addresses the requirement for more VLANs in the virtualized, web-scale datacenter of the future. While not what XVLAN’s authors envisioned, you can now run Layer 3 over Layer 2 over VXLAN over Layer 3 over Layer 2! How meta!
History Lesson From Carrier Networks- PWE3/VPLS and the Intelligent Edge
The evolution of the data center network is reminding me a lot of how carrier backbones evolved in the last two decades. The early days of modern internet backbones were built with IP over ATM. Anyone remember the cell tax? As Performance of IP forwarding caught and surpassed ATM, Packet over SONET emerged as the defacto carrier interface with IP routing and/or MPLS in the control plane. As it became clear IP had won, a new IETF group was formed (PWE3) to define how TDM, Frame Relay, and ATM could be carried over packet networks. This allowed carriers to continue to offer Layer 2 connectivity services to customers but put their dedicated Frame Relay and ATM networks to pasture. Around the same time, VPLS was developed to solve conceptually very similar problems to what VXLAN/VNGRE is solving but in the context of carrier Ethernet services, not data centers.
VXLAN reminds me a lot of PWE3/VPLS in that they both support Layer 2 over Layer 3. Both reflect the fact that Layer 3 and IP is the appropriate convergence layer. In the data center, the Layer 2 service being “offered” is the VLANs to connect and partition VM-to-VM communication.
Further, today’s carrier networks have pushed all the service intelligence and encapsulation to edge routers and attempted to make the packet core as fast and simple as possible. The data center is following the same trend except the new edge of the network is inside the server. The data center core will be focused on simply forwarding packets and light QoS/traffic engineering, similar to what carriers have already done in the IP backbone.
It’s not surprising the data center network is looking more like a carrier network because in the past decade many of the leading IP networking architects and operations folks from the telco carriers have migrated to work at the web scale data centers at Google, Facebook, Yahoo, Microsoft, etc. These folks know how to build large networks, and do it using IP. The web scale networks have had to drive IP to the top of the rack already to work around the limitations of Layer 2 domain size, spanning tree, and to gain the benefits of routing. The drawback of this approach is the limited size of the Layer 2 domain for VM migrations, hence VXLAN.
Open Networking Summit
It’s an exciting time in networking for a lot of reasons. Here’s a handful of changes happening simultaneously that present massive opportunities for emerging companies.
- Unprecedented scale and number of hosts to connect
- New protocols (VXLAN and VNGRE)
- New abstractions (OpenFlow)
- New tiers of switching
- 40/100G coming to data centers soon
- Incumbent vendors showing up over dressed to the data center party creating great opportunities for start-ups.
I will be at the Open Networking Summit (#ONS2011) next week at Stanford. If anyone wants to get together and chat about these or other networking related topics drop me a line at firstname.lastname@example.org or a tweet at @abenik.