6 Network Security Protocols You Should Know

What Are Network Security Protocols?

Network security protocols are network protocols that ensure the integrity and security of data transmitted across network connections. The specific network security protocol used depends on the type of protected data and network connection. Each protocol defines the techniques and procedures required to protect the network data from unauthorized or malicious attempts to read or exfiltrate information.

Related content: Learn more about network security threats security protocols can protect against.

The OSI Network Model

Open Systems Interconnection (OSI) is a reference model for how applications communicate over networks. It shows how each layer of communication is built on top of the other, from the physical wiring to the applications that attempt to communicate with other devices over the network.

The OSI is a reference model that guides technology vendors on the design of interoperable software and hardware, providing a clear framework that describes the capabilities of a network or communications system. For security teams, the OSI model helps understand which layers of the network they need to defend, where specific security threats could strike, and how to prevent and mitigate them.

The OSI Model contains the following layers:

• Layer 1—Physical Layer—the physical cable or wireless connection between network nodes.
• Layer 2—Data Link Layer—creates and terminates connections, breaks up packets into frames and transmits them from source to destination.
• Layer 3—Network Layer—breaks up segments into network packets, and reassembles them upon receipt, and routes packets using an optimal path on the physical network.
• Layer 4—Transport Layer—responsible for reassembling the segments on the receiving end, turning it into data that can be used by the session layer.
• Layer 5—Session Layer—creates communication channels, called sessions, between devices. Keeps sessions open during data transfer and closing them when it ends.
• Layer 6—Presentation Layer—prepares data for the application layer, defining how two devices should encode, encrypt, and compress data to ensure it is received correctly.
• Layer 7—Application Layer—used by end-user software like web browsers and email clients. Sends and receives information that is meaningful for end-users using protocols like HTTP, FTP, and DNS.

6 Types of Network Security Protocols

Following are some of the most common network security protocols. They are arranged by the network layer at which they operate, from bottom to top.

Internet Protocol Security (IPsec) Protocol—OSI Layer 3

IPsec is a protocol and algorithm suite that secures data transferred over public networks like the Internet. The Internet Engineering Task Force (IETF) released the IPsec protocols in the 1990s. They encrypt and authenticate network packets to provide IP layer security. 

IPsec originally contained the ESP and AH protocols. Encapsulating Security Payload (ESP) encrypts data and provides authentication, while Authentication Header (AH) offers anti-replay capabilities and protects data integrity. The suite has since expanded to include the Internet Key Exchange (IKE) protocol, which provides shared keys establishing security associations (SAs). These enable encryption and decryption via a firewall or router. 

IPsec can protect sensitive data and VPNs, providing tunneling to encrypt data transfers. It can encrypt data at the application layer and enables authentication without encryption.

SSL and TLS—OSI Layer 5

The Secure Sockets Layer (SSL) protocol encrypts data, authenticates data origins, and ensures message integrity. It uses X.509 certificates for client and server authentication. SSL authenticates the server with a handshake, negotiating security session parameters and generating session keys. It can then securely transmit the data by authenticating its origin.

SSL sessions use cryptographic algorithms similar to the algorithms used by the client and server (determined during the handshake). Servers may support encryption with algorithms like AES and Triple DES. 

X.509 server certificates are a requirement for SSL, enabling the client to validate the server. SSL can also use X.509 client certificates for authentication. These certificates must be signed by a trusted certificate authority in the server’s keyring. 

Transport Layer Security (TLS) is an SSL-based protocol defined by the IETF (SSL is not).

Datagram Transport Layer Security (DTLS)—OSI Layer 5

DTLS is a datagram communication security protocol based on TLS. It does not guarantee message delivery or that messages arrive in order. DTLS introduces the advantages of datagram protocols, including lower latency and reduced overhead.

Kerberos Protocol—OSI Layer 7

Kerberos is a service request authentication protocol for untrusted networks like the public Internet. It authenticates requests between trusted hosts, offering built-in Windows, Mac, and Linux operating system support. 

Windows uses Kerberos as its default authentication protocol and a key component of services like Active Directory (AD). Broadband service providers use it to authenticate set-top boxes and cable modems accessing their networks.

Systems, services, and users, only need to trust the KDC when using Kerberos. KDC offers authentication and grants tickets to enable nodes to authenticate each other. Kerberos uses shared secret cryptography to authenticate packets and protect them during transmission. 

Simple Network Management Protocol (SNMP)—OSI Layer 7

SNMP is a network device management and monitoring protocol that works at the application layer. It can secure devices on LANs or WANs. SNMP provides a shared language to allow devices like servers and routers to communicate via a network management system. SNMP is an original part of the Internet protocol suite defined by the IETF. 

Components of the SNMP architecture include a manager, an agent, and a management information base (MIB). The manager is the client, the agent is the server, and the MIB is the database. The SNMP agent responds to the manager’s requests using the MIB. While SNMP is widely available, administrators must adjust the default settings to enable communication between the agents and the network management system to implement the protocol. 

With the introduction of SNMPv3 in in 2004, the SNMP protocol gained three important security features: encryption of packets to prevent eavesdropping, integrity checks to ensure packets were not been tampered in transit, and authentication to verify that communications come from a known source.

HTTP and HTTPS—OSI Layer 7

HTTP is an application protocol that specifies rules for web file transfers. Users indirectly use HTTP when they open their web browser. It runs on top of the Internet protocol suite. 

HTTPS is the secure version of HTTP, securing the communication between browsers and websites. It helps prevent DNS spoofing and man-in-the-middle attacks, which is important for websites that transmit or receive sensitive information. All websites requiring user logins or handling financial transactions are attractive data theft targets and should be using HTTPS.

HTTPS runs over the SSL or TLS protocol using public keys to enable shared data encryption. HTTP uses port 80 by default, while HTTPS uses port 443 for secure transfers. With HTTPS, the server and browser must establish the communication parameters before initiating data transfers.