Best Practices for Using Multiple Network Interfaces (NICs) with NI Products

Before explaining best practices for configuring controllers with multiple NICs, it is necessary to understand some basic concepts related to Ethernet networking. This section explains how individual NICs are addressed, how packets are routed on Ethernet networks, and what a typical Ethernet packet transfer looks like. 

If you are already familiar with these concepts or are not concerned about the “why” behind the recommendations in this paper, then you can skip to the Guidelines for Configuring Multi-NIC Systems section.

Data Link Layer: MAC Addressing

Just like a home or business address must be specified in order to send a piece of postal mail, every port on every Ethernet adapter is assigned a unique address called a Medium Access Control (MAC) address by the manufacturer. MAC addresses consist of 48 bits, and are commonly expressed in groups of 2 hexadecimal digits (e.g. 01:23:45:67:89:ab). The MAC address or addresses for a specific Ethernet adapter are commonly printed on a label on the device itself, and are also accessible via operating system networking settings. 

When Ethernet data is sent between two computers, it is sent in small pieces called “packets”, and those packets contain the MAC address of the receiving interface. Each interface is continually listening for packets sent to its unique address.

Figure 1. Each network interface controller (NIC) in a computer has a unique medium access control (MAC) address. If multiple NICs are installed in one computer, they each have their own MAC addresses.

Internet Layer Addressing

If every computer in the world were connected to one large wire and knew every other computer’s MAC address, then no further addressing schemes would be required. However, this is impractical for several reasons. First, with such a large shared connection packet collisions would occur frequently, causing communication to be unreliable. Second, storing the address of every other computer on the planet on each machine would require a large amount of memory, and is not a good strategy due to the constant addition of new devices.

IP Addresses (IPv4)

Clearly, a different scheme of addressing is needed to enable computers to communicate with each other via Ethernet. In practice, users can assign one or more Internet Protocol (IP) addresses to each MAC address, and set these IP addresses up in such a way that most network traffic occurs in small “pools” rather than across a wider group of machines. In practice, IP addresses are typically specified by a 32-bit value, and are commonly written as four groups of 3-digit decimal values (e.g. 192.168.100.100). Consider a network with 10 computers configured such that the computers have IP addresses ranging from 192.168.100.0 to 192.168.100.9. If all machines follow a rule where all addresses with format 192.168.100.xxx are assumed to be inside the internal network, then it is easy to tell whether communicating with a given IP address requires accessing some other network or not.

Sample rule: all interfaces with IP addresses in the range 192.168.100.0 to 192.168.100.255 are inside of the local network, and interfaces with other addresses are outside.

IP Address
Inside or Outside Local Network?
(following the sample rule)

192.168.100.6
Inside

192.168.100.104
Inside

192.169.100.100
Outside

192.169.101.255
Outside

Table 1. Since IP addresses can be assigned to individual NICs by the user, it is possible to assign similar IP addresses to a group of interfaces and then easily determine whether a given IP address is inside or outside of the local network.

This simple idea has significant benefits. Whereas MAC addresses do not typically follow a pattern in a given set of machines (they are randomly distributed by vendor), since IP addresses are user-configurable they can be set up to follow a logical, hierarchical pattern. Using this pattern, small groups of computers can communicate with each other with only local traffic, therefore reducing data collisions in the larger network. In addition, since there are more than 2^32 connected Ethernet devices on the planet, IP addresses can be reused across multiple smaller networks while those machines are still accessible via a process called Network Address Translation (NAT). A detailed discussion of NAT is beyond the scope of this tutorial.

When configuring IP addresses, you are likely to run across addresses of the form 10.x.x.x, 192.168.x.x, or 172.x.x.x. Each of these ranges have been specially designated for private network use, so you can be sure that no other public server (e.g. google.com) will use the same IP address if you use one of these.

It should be noted that the information in this section pertains to the most commonly used version of the Internet Protocol called IPv4. The next version of the Internet Protocol, IPv6, uses 128-bit addresses but is not widely used today (although adoption is expected to grow in the next several years).

Subnet Masks

So, when looking at an IP address to send data to, how does a given computer decide if it is inside of the local network, or whether it will rely accessing an outside network? In other words, if we want computers to follow a rule as was the case with the example in the previous section, how is that rule specified?

The key component here is the subnet mask, a group of 32 bits expressed in 4 groups of numbers just like the IP address (e.g. 255.255.255.192). A binary 1 value in one of the 32 bit positions means that if the IP address bit is different in this position between the sender and receiver, then Ethernet packets must be routed to an outside network. On the contrary, a binary zero in a given position means that if the IP address bit is different then it does not matter; if all other bits are the same between the sender and receiver’s IP address then the receiver is assumed to be on the same local network.

Figure 2. Subnet masks provide a way to specify the “rule” for which IP addresses are included in a local network and which fall outside of the local network.

Gateways and Default Gateways

How are Ethernet packets passed between local and outside networks? This task is typically performed by a device called a router, that is also often referred to as a network gateway (in this tutorial the two terms will be used interchangeably). For each Ethernet adapter connected to a computer, the user often specifies an IP address, subnet mask, and default gateway. The default gateway is the address of the router that should be used when packets must be sent outside of the local network.

ARP Requests

At this point, note that at the lowest level, every single Ethernet packet that is sent must be sent to a MAC address. IP addresses provide some advantages including the use of subnets, but keep in mind that simply knowing the IP address of a remote computer’s interface is not enough information to ensure that a packet reaches that computer. For example, to communicate between two computers on remote networks it may be necessary to pass through several routers. At a low-level, one packet must be sent between the sending computer and the first router, then another packet from the first router to the next, and so forth until the final packet finally reaches the destination computer. For the packet to be sent between each link, knowing the MAC address of the receiving router or computer’s interface is critical.

Imagine starting with an IP address of a destination computer on the local network: how does the originating computer obtain the correct MAC address to send packets to? The Address Resolution Protocol (ARP) is used for this. Continuing with the example, the sending computer will send a special type of broadcast message called an ARP request on the local network (requesting the MAC address for a certain IP address). All Ethernet devices on the network are continually listening for these special requests, and will respond back to the sender with their MAC address if they match the IP address specified. Once the MAC address is received by the sending computer, that computer will usually store the MAC address / IP address pair in a cache table for later usage. This prevents the continual need to send ARP requests for the same destination interface.

Figure 3. For a computer to send data to a remote Ethernet interface, it must know the MAC address of that interface. In the case that the MAC address is not known, a computer can send an Address Resolution Protocol (ARP) request to a specified IP address, and the destination machine will reply back with its MAC address.

If a router on the network knows that the IP requested is external to the local network, then it can also respond to the ARP request with its own MAC address. This is called “proxy ARP”.

Summary: Anatomy of an Ethernet Packet Transfer

Let us walk through the (simplified) process of sending a single Ethernet packet from a source PC to a destination PC. We will assume that the Source PC contains multiple Ethernet interfaces.

1. An Ethernet packet is ready to be sent by the source machine (this could be associated with a specific application or OS process). The packet contains information about the IP address of the receiver (this may have been obtained from a Domain Name Service (DNS) server).
2. If there is already a cached ARP entry containing the MAC address corresponding to the receiver’s IP address, then proceed to step 4.
3. The Sender looks at the IP address of the packet, and the subnet mask of its Ethernet interfaces.
   1. If the IP address falls within the subnet of a specific interface, then an ARP request is sent from that interface.
   2. If the IP address falls outside of the subnet of any interfaces, then the sender looks at the default gateway entry for each interface.
      1. If there is only one default gateway entry for all interfaces, then use its MAC address as contained in the ARP table or send a new ARP request to find the MAC address of the gateway.
      2. If there are multiple default gateway entries, then use the one with the lowest metric (determined by several factors depending on the OS used). Find its MAC address in the ARP table, or send a new ARP request to find the MAC address of the gateway.
      3. If there are no default gateway entries, send an ARP request on one or more interfaces to see if a gateway responds with its own MAC (proxy ARP).
4. Send the Ethernet packet using the MAC address from the ARP cache or ARP request. If the ARP request does not produce a result, then the packet is not deliverable. If the receiver is on a remote subnet, then the packet is sent to a router and it is the router’s job to send the packet over the next link so that it ultimately reaches the receiver.