Bridging Overview – Oracle Solaris Administration: Network Interfaces and Network Virtualization

Bridging Overview

Bridges are used to connect separate network segments. When connected by a bridge,
the attached network segments communicate as if they were a single network segment.
Bridging is implemented at the datalink layer (L2) of the networking stack. Bridges
use a packet-forwarding mechanism to connect subnetworks together.

While bridging and routing can both be used to distribute information about the
locations of resources on the network, they differ in several ways. Routing is
implemented at the IP layer (L3) and uses routing protocols. No routing protocols
are used on the datalink layer. Instead, the destinations of forwarded packets are
determined by examining the network traffic that is received on the links that
are attached to the bridge.

When a packet is received, its source address is examined. The packet’s source
address associates the node from which the packet was sent to the
link on which it is received. Thereafter, when a received packet uses that
same address as the destination address, the bridge forwards the packet over the link
to that address.

The link associated with a source address might be an intermediate link that
is connected to another bridge in the bridged subnetwork. Over time, all of
the bridges within the bridged subnetwork “learn” which of the links sends a
packet toward a given node. Thus, the packet’s destination address is used to
direct the packet to its final destination by means of hop-by-hop bridging.

A local “link-down” notification indicates that all nodes on a given link are
no longer reachable. In this situation, packet forwarding to the link is halted
and all forwarding entries over the link are flushed. Forwarding entries are also
aged away over time. When a link is restored, packets that are received
over the link are treated as new. The “learning” process based on a
packet’s source address begins again. This process enables the bridge to properly forward
packets over that link when the address is used as the destination address.

To forward packets to their destinations, bridges must listen in promiscuous mode on
every link that is attached to the bridge. Listening in promiscuous mode causes
bridges to become vulnerable to the occurrences of forwarding loops, in which packets
circle forever at full line rate. Thus, bridging uses the Spanning Tree Protocol
(STP) mechanism to prevent network loops that would render the subnetworks unusable.

In addition to using STP and the Rapid Spanning Tree Protocol (RSTP) for
bridges, Oracle Solaris supports the TRILL protection enhancement. STP is used by default,
but you can use TRILL by specifying the -P trill option for the bridging
commands.

Using a bridge configuration simplifies the administration of the various nodes in the
network by connecting them into a single network. By connecting these segments through
a bridge, all the nodes share a single broadcast network. Thus, each node
can reach the others by using network protocols such as IP rather than
by using routers to forward traffic across network segments. If you do not
use a bridge, you must configure IP routing to permit the forwarding of
IP traffic between nodes.

The following figure shows a simple bridged network configuration. The bridge, goldengate, is
an Oracle Solaris system that has bridging configured. sanfrancisco and sausalito are systems that
are physically connected to the bridge. Network A uses a hub that is
physically connected to the bridge on one side and to computer systems on
the other side. The bridge ports are links, such as bge0, bge1,
and bge2.

Figure 11-1 Simple Bridged Network

Bridge networks can be formed into rings that physically connect several bridges together.
Such configurations are common in networks. This type of configuration could cause problems
with old packets saturating the network links by endlessly looping around the ring.
To protect against such looping conditions, Oracle Solaris bridges implement both the STP
and TRILL protocols. Note that most hardware bridges also implement STP loop protection.

The following figure shows a bridged network that is configured in a ring.
The configuration shows three bridges. Two systems are physically connected to westminster. One
system is physically connected to waterloo. And one system is physically connected
to tower. Each of the bridges are physically connected to each other through
the bridge ports.

When STP or RSTP is used for loop protection, the physical loop
is mitigated by preventing one of the connections in the loop from forwarding
packets. The figure shows that the physical link between the westminster and tower
bridges is not used to forward packets.

Note that by shutting down usable physical links to perform loop protection, STP
and RSTP cost you bandwidth.

Unlike STP and RSTP, TRILL does not shut down physical links to
prevent loops. Instead, TRILL computes the shortest-path information for each TRILL node in the
network and uses that information to forward packets to individual destinations.

As a result, TRILL enables the system to leave all links in use
at all times. Loops are not a problem as they are handled
similarly to the way that IP handles loops. Namely, TRILL creates routes as
needed and uses forwarding hop limits to avoid problems that are caused by
momentary loop states.

Figure 11-2 Bridged Network Ring

Caution – Do not set local-mac-address?=false on SPARC platforms, or the systems will errantly use
the same MAC address on multiple ports and on the same network.

Note – Do not configure a link into a bridge when the highest possible levels of
performance are required. Bridging requires that the underlying interfaces are in promiscuous
mode, which disables a number of important optimizations that are in the hardware,
driver, and other layers of the system. The disabling of these performance enhancements is
an unavoidable consequence of the bridging mechanism.

You can use a bridge on a system where some of the system’s
links are not bridged and are thus not subject to those constraints. These
performance issues only affect those links that are configured to be part of
a bridge.

For information about STP, see IEEE 802.1D-1998. For information about RSTP, see IEEE
820.1Q-2004. For information about TRILL, see the Internet Engineering Task Force (IETF) TRILL draft documents.

Link Properties

These link properties can be shown and modified by the dladm show-linkprop, dladm set-linkprop,
and reset-linkprop commands:

default_tag

Define the default virtual local area network (VLAN) ID for untagged packets that are sent to and from the link. Valid values are from 0 to 4094. The default value is 1. Only non-VLAN and non-virtual network interface card (VNIC) type links have this property. Setting this value to 0 disables the forwarding of untagged packets to and from the port. (This is a MAC property.)

Note – This property is also used outside the scope of bridging to specify the IEEE Port VLAN Identifier (PVID) for the link. When default_tag is non-zero, you cannot create a VLAN that has that same ID on the link because the base link itself automatically represents the PVID.

For example, if PVID is set to 5 on net0, you cannot create a VLAN with ID 5 on net0. To specify VLAN 5 in this situation, use net0.

You cannot set default_tag to be equal to the ID of any existing VLAN that is created on that link. For instance, the following command creates VLAN 22 on net0:

# dladm create-vlan -l net0 -v 22 myvlan0

In this situation, you cannot set default_tag to 22, which would make both net0 and myvlan0 represent the same VLAN.

By setting default_tag to 0, you enable untagged packets on net0 to be unassociated with any VLAN at all. This situation prevents such packets from being forwarded by a configured bridge.

forward

Enable and disable traffic forwarding through the bridge. This property exists on all links except for VNIC links. Valid values are 1 (true) and 0 (false). The default value is 1. When disabled, a VLAN that is associated with a link instance will not forward traffic through the bridge. Disabling forwarding is equivalent to removing the VLAN from the “allowed set” for a traditional bridge. This means that VLAN-based I/O to the underlying link from local clients continues, but no bridge-based forwarding is performed.

stp

Enable and disable STP and RSTP. Valid values are 1 (true) and 0 (false). The default value is 1, which enables STP and RSTP. When set to 0, the link does not use any type of Spanning Tree Protocol and is placed into forwarding mode at all times. The forwarding mode uses bridge protocol data unit (BPDU) guarding. Disable STP and RSTP when you want to configure point-to-point links that are connected to end nodes. Only non-VLAN and non-VNIC type links have this property.

stp_cost

Represent STP and RSTP cost values for using the link. Valid values are from 1 to 65535. The default value is 0, which is used to signal that cost is automatically computed by link type. The following values represent the cost for several link types: 100 for 10 Mbps, 19 for 100 Mbps, 4 for 1 Gbps, and 2 for 10 Gbps.

stp_edge

Specify whether the port is connected to other bridges. Valid values are 1 (true) and 0 (false). The default value is 1. If set to 0, the daemon assumes that the port is connected to other bridges even if no BPDUs of any type are seen.

stp_p2p

Specify the connection mode type. Valid values are true, false, and auto. The default value is auto, which automatically discovers point-to-point connections. Specify true to force to point-to-point mode, and specify false to force normal multipoint mode.

stp_priority

Set the STP and RSTP port priority value. Valid values are from 0 to 255. The default value is 128. The STP and RSTP port priority value is used to determine the preferred root port of a bridge by prepending the value to the port identifier. The lower the numerical value is, the higher the priority.

STP Daemon

Each bridge that you create by using the dladm create-bridge command is represented as
an identically named SMF instance of svc:/network/bridge. Each instance runs a copy
of the /usr/lib/bridged daemon, which implements the STP.

The following command example creates a bridge called pontevecchio:

# dladm create-bridge pontevecchio

The system creates an SMF service called svc:/network/bridge:pontevecchio and an observability node
called /dev/net/pontevecchio0.

For safety purposes, all ports run standard STP by default. A bridge that
does not run some form of bridging protocol, such as STP, can
form long-lasting forwarding loops in the network. Because Ethernet has no hop-count or
TTL on packets, any such loops are fatal to the network.

When you know that a particular port is not connected to another
bridge (for example, a direct point-to-point connection to a host system), you can administratively
disable STP for that port. Even if all ports on a bridge have
STP disabled, the STP daemon still runs. The daemon continues to run for
the following reasons:

• To handle any new ports that are added

• To implement BPDU guarding

• To enable or disable forwarding on the ports, as necessary

When a port has STP disabled, the bridged daemon continues to listen for
BPDUs (BPDU guarding). The daemon uses syslog to flag any errors and disables
forwarding on the port to indicate a serious network misconfiguration. The link is
reenabled when link status goes down and comes up again, or when you
manually remove the link and re-add it.

If you disable the SMF service instance for a bridge, bridge forwarding stops
on those ports as the STP daemon is stopped. If the instance
is restarted, STP starts from its initial state.

TRILL Daemon

Each bridge that you create by using the dladm create-bridge -P trill command is represented
as an identically named SMF instance of svc:/network/bridge and svc:/network/routing/trill. Each instance of
svc:/network/routing/trill runs a copy of the /usr/lib/trilld daemon, which implements the TRILL protocol.

The following command example creates a bridge called bridgeofsighs:

# dladm create-bridge -P trill bridgeofsighs

The system creates two SMF services called svc:/network/bridge:bridgeofsighs and svc:/network/routing/trill:bridgeofsighs. In addition, the
system creates an observability node called /dev/net/bridgeofsighs0.

Debugging Bridges

Each bridge instance is assigned an “observability node,” which appears in the /dev/net/
directory and is named by the bridge name plus a trailing 0.

The observability node is intended for use with the snoop and wireshark
utilities. This node behaves like a standard Ethernet interface, except for the transmission
of packets, which are silently dropped. You cannot plumb IP on top of
an observability node, and you cannot perform bind requests (DL_BIND_REQ) unless you use
the passive option.

When used, the observability node makes a single unmodified copy of every packet
handled by the bridge available to the user. This behavior is similar to
a “monitoring” port on a traditional bridge, and is subject to the usual
DLPI “promiscuous mode” rules. You can use pfmod or features in the
snoop and wireshark utilities to filter based on VLAN ID.

The delivered packets represent the data that is received by the bridge.

Caution – In the cases where the bridging process adds, removes, or modifies a VLAN
tag, the data shown describes the state prior to this process taking place.
This rare situation might be confusing if there are distinct default_tag values
used on different links.

To see the packets that are transmitted and received on a particular link
(after the bridging process is complete), run snoop on the individual links rather
than on the bridge observability node.

For information about observability nodes, see Observability Features for Network Virtualization and Resource Control.

Other Bridge Behaviors

The following sections describe how link behavior changes when bridges are used in
the configuration.

For information about standard link behavior, see Administering Virtual Local Area Networks.

DLPI Behavior

The following describes the differences in link behavior when a bridge is enabled:

• Link up (DL_NOTE_LINK_UP) and link down (DL_NOTE_LINK_DOWN) notifications are delivered in the aggregate. This means that when all external links are showing link-down status, the upper-level clients that are using the MAC layers will also see link-down events. When any external link on the bridge shows link-up status, all upper-level clients see link-up.

  This aggregate link-up and link-down reporting is performed for the following reasons:

  • When link-down is seen, nodes on the link are no longer reachable. This is not true when the bridging code can still send and receive packets through another link. Administrative applications that need the actual status of links can use the existing MAC-layer kernel statistics to reveal the status. These applications are unlike ordinary clients, such as IP, in that they report hardware status information and do not get involved in forwarding.

  • When all external links are down, the status shows through as though the bridge itself were shut down. In this special case, the system recognizes that nothing could possibly be reachable. The trade-off is that bridges cannot be used to allow local-only communication in the case where all interfaces are “real” (not virtual) and all are disconnected.

• All link-specific features are made generic. Links that support special hardware acceleration features are unable to use those features because actual output link determination is not made entirely by the client. The bridge forwarding function must choose an output link based on the destination MAC address, and this output link can be any link on the bridge.

VLAN Administration

By default, VLANs that are configured on the system are forwarded among all
the ports on a bridge instance. When you invoke the dladm create-vlan or
dladm create-vnic -v command, and the underlying link is part of a bridge, that command
will also enable forwarding of the specified VLAN on that bridge link.

To configure a VLAN on a link and disable forwarding to or
from other links on the bridge, you must disable forwarding by setting the
forward property with the dladm set-linkprop command.

Use the dladm create-vlan command to automatically enable the VLAN for bridging when the
underlying link is configured as part of a bridge.

VLANs are ignored in the standards-compliant STP. The bridging protocol computes just one
loop-free topology by using tag-free BPDU messages, and uses this tree to enable
and disable links. You must configure any duplicate links that are provisioned in
your networks such that when those links are automatically disabled by STP, the
configured VLANs are not disconnected. This means that you should either run all
VLANs everywhere on your bridged backbone or carefully examine all redundant links.

TRILL does not need to follow the complex STP rules. Instead, TRILL automatically
encapsulates packets that have the VLAN tag intact, and passes them through the
network. This means that TRILL binds together isolated VLANs where the same VLAN
ID has been reused within a single bridged network.

This is an important difference from STP where you might reuse VLAN tags
in isolated sections of the network to manage sets of VLANs that
are larger than the 4094 limit. While you cannot use TRILL to manage
networks in this way, you might be able to implement other solutions, such
as provider-based VLANs.

In an STP network with VLANs, it might be difficult to configure
the failover characteristics to prevent VLAN partitioning when STP disables the “wrong” link. The
relatively small loss of functionality in isolated VLANs is more than made up
for in the robustness of the TRILL model.

VLAN Behavior

The bridge performs forwarding by examining the allowed set of VLANs and the
default_tag property for each link. The general process is as follows:

• Input VLAN determination. This task begins when a packet is received on a link. When a packet is received, it is checked for a VLAN tag. If that tag is not present or if the tag is priority-only (tag zero), the default_tag configured on that link (if not set to zero) is taken as the internal VLAN tag. If the tag is not present or zero and default_tag is zero, the packet is ignored. No untagged forwarding is performed. If the tag is present and is equal to default_tag, the packet is also ignored. Otherwise, the input tag is taken to be the input VLAN.

• Link membership check. If the input VLAN is not configured as an allowed VLAN on this link, the packet is ignored. Forwarding is then computed, and the same check is made for the output link.

• Tag update. If the VLAN (nonzero at this point) is equal to default_tag on the output link, the tag on the packet (if any) is removed, regardless of priority. If the VLAN is not equal to default_tag on the output link, a tag is added if not currently present, and the tag is set for the output packet, with the current priority copied into the packet.

Note – In the case where forwarding sends to multiple interfaces (for broadcast, multicast, and
unknown destinations), the output link check and tag update must be done independently
for each output link. Some transmissions might be tagged while others are untagged.

Bridge Configuration Examples

The following examples show how to view information about bridge configurations and bridging
services.

• You can obtain information about bridges by running the following command:

  # dladm show-bridge
  BRIDGE      PROTECT ADDRESS            PRIORITY DESROOT
  tonowhere   trill   32768/66:ca:b0:39:31:5d 32768 32768/66:ca:b0:39:31:5d
  sanluisrey  stp     32768/ee:2:63:ed:41:94 32768 32768/ee:2:63:ed:41:94
  pontoon     trill   32768/56:db:46:be:b9:62 32768 32768/56:db:46:be:b9:62

• You can obtain TRILL nickname information for a bridge by running the following command:

  # dladm show-bridge -t tonowhere
  NICK FLAGS LINK        NEXTHOP
  38628 --   simblue2    56:db:46:be:b9:62
  58753 L    --          --