Some workloads require the use of VLAN interfaces in virtual machines. VMware terms this feature “Virtual Guest Tagging” or “VLAN Guest Tagging” while OpenStack calls it “VLAN-aware instances”. See how OpenShift Virtualization can pass 802.1q trunks to VMs using a traditional Linux Bridge interface.

Bridges and VLANs

Traditionally a bridge simply existed to relay data link layer ethernet frames (just think packets) from one physical segment to another. A switch is an evolved bridge which can apply more intelligence when deciding which interface to send a frame.

Each frame has a source and destination MAC address. Switches collect the MAC address from frames and understand which port a MAC address was learned on, this enables more efficient decisions when directing frames to ports. Additionally switches can be segmented into multiple broadcast domains though the use of VLANs by tagging frames with a unique ID.

The uplink feeding data to a switch may be an 802.1q trunk which will include a VLAN ID tag on each frame. The tags remain on the frame until it is passed out a port which is configured as an “access port” associated with a specific VLAN. If a tag comes into the switch via that access port, the tag assocated with that port is added back to the frame by the switch.

That’s normally exactly what you want, but let’s dive into a solution to treate the port as a trunk and keep tags attached and visible inside of the VM operating system.

First a little background on OpenShift Networking.

OVS Bridge and Linux Bridge

OpenShift uses the OVN-Kubernetes CNI which includes support for a localnet topology layered upon an OVS Bridge. See this previous post on OVN. While OVN is the most featureful and recommended interface, it does not yet support VLAN tags through a localnet attachment.

Fortunately this can be accomplished using a Linux Bridge interface. Despite the name, a Linux Bridge has intelligence for handling frame construction and the ports attached to it like a switch.

Above is a diagram of a node having 3 bridges. Bridge br-ex is the default management interface, the second br-vmdata was created to attach VMs to provider networks as localnet secondary networks. Both of these bridges are OVS Bridges. The third bridge br-trunk is a Linux Bridge and was created only for the use case we are discussing here.

Creating the Linux Bridge

It’s important to note that a host NIC can not be assigned to both an OVS Bridge and a Linux Bridge at the same time, so a dedicated host NIC will be required.

Here we create a Linux bridge called br-trunk on a dedicated NIC called ens256.

📓 NodeNetworkConfigurationPolicy to create Linux Bridge br-trunk If ens256 is not in every host you’ll need to add add a NodeSelector. ref

• br-trunk nncp.yaml

   1---
   2apiVersion: nmstate.io/v1
   3kind: NodeNetworkConfigurationPolicy
   4metadata:
   5  name: br-trunk
   6spec:
   7  desiredState:
   8    interfaces:
   9      - name: ens256
  10        ipv4:
  11          enabled: false
  12        ipv6:
  13          enabled: false
  14        state: up
  15        type: ethernet
  16
  17      - name: br-trunk
  18        description: |-
  19          Linux Bridge with ens256 as a port
  20          allowing all VLANs and untagged traffic.          
  21        type: linux-bridge
  22        state: up
  23        bridge:
  24          options:
  25            stp:
  26              enabled: false
  27          port:
  28            - name: ens256
  29              vlan: {}
  

Next create a network attachment definition called trunk in the demo-vgt namespace. It will be a cnv-bridge type and reference the br-trunk interface just created above.

📓 NetworkAttachmentDefinition to form a an 802.1q trunk from a Linux Bridge to VM ref

• trunk nad.yaml

   1---
   2apiVersion: k8s.cni.cncf.io/v1
   3kind: NetworkAttachmentDefinition
   4metadata:
   5  annotations:
   6    description: 802.1q Trunk Connection
   7  name: trunk
   8  namespace: demo-vgt
   9spec:
  10  # omitting vlanId on Linux Bridge results in all VLANs passed
  11  # omitting vlan on OVS Bridge results in only VLAN 0 (native) passing
  12  config: |-
  13    {
  14      "cniVersion": "0.4.0",
  15      "name": "trunk",
  16      "type": "cnv-bridge",
  17      "macspoof": false,
  18      "bridge": "br-trunk",
  19      "netAttachDefName": "demo-vgt/trunk",
  20      "vlanId": {},
  21      "ipam": {}
  22    }    
  

Demos

Host to Guest VLAN Tagging

In this demo, the provider (meaning existing outside the OpenShift cluster) VLAN 1924 is among several VLAN tags trunked to ens256 on the OpenShift Node. You will see a VLAN interface created in the VM called eth1.1924 which is directly attached to this physical VLAN.

Demo (source): 802.1q trunk to a VM using linux-bridge on the host and cnv-bridge type NetworkAttachmentDefinition

📺 ASCII Screencast

Setup bridge & network attachment, create VMs, test networking, and cleanup.

Guest to Guest VLAN Tagging

When the VM NIC is attached via Linux Bridge it also becomes possible to make up arbitrary VLAN tags to trunk between virtual machines. In otherwords you can make up your own VLAN tags and trunk them peer-to-peer between VMs.

This demo shows made up VLANs 222 and 333 being created from thin air and passed between 2 VMs via the same trunk attachment to the br-trunk Linux Bridge used above.

Demo (source): 802.1q trunk between VMs using linux-bridge on the host and cnv-bridge type NetworkAttachmentDefinition

• VM network-setup.sh

  # create eth1.222 and eth1.333 VLAN interfaces
  # use .1 IP on first VM and .2 IP on second VM to enable ICMP test
  NIC=eth1
  VLAN=222
  
  nmcli -f connection.id con show $NIC.$VLAN 2>&1 > /dev/null
  
  if [ $? -ne 0 ]; then
    nmcli connection add type vlan \
      con-name $NIC.$VLAN \
      ifname $NIC.$VLAN \
      vlan.parent $NIC \
      vlan.id $VLAN
  fi
  
  nmcli connection modify $NIC.$VLAN \
    ipv4.route-metric 222 \
    ipv4.method static \
    ipv4.address 10.2.2.1/24
  
  nmcli connection up $NIC.$VLAN
  
  VLAN=333
  
  nmcli -f connection.id con show $NIC.$VLAN 2>&1 > /dev/null
  
  if [ $? -ne 0 ]; then
    nmcli connection add type vlan \
      con-name $NIC.$VLAN \
      ifname $NIC.$VLAN \
      vlan.parent $NIC \
      vlan.id $VLAN
  fi
  
  nmcli connection modify $NIC.$VLAN \
    ipv4.route-metric 333 \
    ipv4.method static \
    ipv4.address 10.3.3.1/24
  
  nmcli connection up $NIC.$VLAN
  

📺 ASCII Screencast

Demonstrate peer to peer VLAN trunking

Summary

OVS Bridging + localnet topology is the recommended way to attach a OpenShift Virtualization VM guest to a provider VLAN. However, when there is a need to pass VLAN tags all the way through to the VM guest your options become special SR-IOV hardware and possibly QinQ or a software Linux Bridge. You just saw how to implement the Linux Bridge solution.

Stand by for OpenShift’s User Defined Networks feature which is just around the corner. UDN will bring greater flexibility, simplicity, tenancy, and enhanced support for “bringing your own netork”. However, VLAN tags will not be supported out of the gate.

More on UDN in a future post.

References

• Demo Source
• Container Network Interface Specification
• OVN-Kubernetes CNI Plugin - github.com
• OVN-Kubernetes
• RFE Support Trunking on OVN-K8s Localnet
• Linux Bridge
• GUI Free Life OVN Inspection post
• OpenStack VLAN Aware Instances