What is IPsec (Internet Protocol Security)?

IPsec is a group of protocols that are used together to set up encrypted connections between devices. It helps keep data sent over public networks secure. IPsec is often used to set up VPNs, and it works by encrypting IP packets, along with authenticating the source where the packets come from.

Within the term “IPsec,” “IP” stands for “Internet Protocol” and “sec” for “secure.” The Internet Protocol is the main routing protocol used on the Internet; it designates where data will go using IP addresses. IPsec is secure because it adds encryption* and authentication to this process.

Encryption is the process of concealing information by mathematically altering data so that it appears random. In simpler terms, encryption is the use of a “secret code” that only authorized parties can interpret.

What is a VPN? What is an IPsec VPN?

A virtual private network (VPN) is an encrypted connection between two or more computers. VPN connections take place over public networks, but the data exchanged over the VPN is still private because it is encrypted.

VPNs make it possible to securely access and exchange confidential data over shared network infrastructure, such as the public Internet. For instance, when employees are working remotely instead of in the office, they often use VPNs to access corporate files and applications.

Many VPNs use the IPsec protocol suite to establish and run these encrypted connections. However, not all VPNs use IPsec. Another protocol for VPNs is SSL/TLS, which operates at a different layer in the OSI model than IPsec. (The OSI model is an abstract representation of the processes that make the Internet work.)

How do users connect to an IPsec VPN?

Users can access an IPsec VPN by logging into a VPN application, or “client.” This typically requires the user to have installed the application on their device.

VPN logins are usually password-based. While data sent over a VPN is encrypted, if user passwords are compromised, attackers can log into the VPN and steal this encrypted data. Using two-factor authentication (2FA) can strengthen IPsec VPN security, since stealing a password alone will no longer give an attacker access.

How does IPsec work?

IPsec connections include the following steps:

Key exchange: Keys are necessary for encryption; a key is a string of random characters that can be used to “lock” (encrypt) and “unlock” (decrypt) messages. IPsec sets up keys with a key exchange between the connected devices, so that each device can decrypt the other device’s messages.

Packet headers and trailers: All data that is sent over a network is broken down into smaller pieces called packets. Packets contain both a payload, or the actual data being sent, and headers, or information about that data so that computers receiving the packets know what to do with them. IPsec adds several headers to data packets containing authentication and encryption information. IPsec also adds trailers, which go after each packet’s payload instead of before.

Authentication: IPsec provides authentication for each packet, like a stamp of authenticity on a collectible item. This ensures that packets are from a trusted source and not an attacker.

Encryption: IPsec encrypts the payloads within each packet and each packet’s IP header (unless transport mode is used instead of tunnel mode — see below). This keeps data sent over IPsec secure and private.

Transmission: Encrypted IPsec packets travel across one or more networks to their destination using a transport protocol. At this stage, IPsec traffic differs from regular IP traffic in that it most often uses UDP as its transport protocol, rather than TCP. TCP, the Transmission Control Protocol, sets up dedicated connections between devices and ensures that all packets arrive. UDP, the User Datagram Protocol, does not set up these dedicated connections. IPsec uses UDP because this allows IPsec packets to get through firewalls.

Decryption: At the other end of the communication, the packets are decrypted, and applications (e.g. a browser) can now use the delivered data.

What protocols are used in IPsec?

In networking, a protocol is a specified way of formatting data so that any networked computer can interpret the data. IPsec is not one protocol, but a suite of protocols. The following protocols make up the IPsec suite:

Authentication Header (AH): The AH protocol ensures that data packets are from a trusted source and that the data has not been tampered with, like a tamper-proof seal on a consumer product. These headers do not provide any encryption; they do not help conceal the data from attackers.

Encapsulating Security Protocol (ESP): ESP encrypts the IP header and the payload for each packet — unless transport mode is used, in which case it only encrypts the payload. ESP adds its own header and a trailer to each data packet.

Security Association (SA): SA refers to a number of protocols used for negotiating encryption keys and algorithms. One of the most common SA protocols is Internet Key Exchange (IKE).

Finally, while the Internet Protocol (IP) is not part of the IPsec suite, IPsec runs directly on top of IP.

What is the difference between IPsec tunnel mode and IPsec transport mode?

IPsec tunnel mode is used between two dedicated routers, with each router acting as one end of a virtual “tunnel” through a public network. In IPsec tunnel mode, the original IP header containing the final destination of the packet is encrypted, in addition to the packet payload. To tell intermediary routers where to forward the packets, IPsec adds a new IP header. At each end of the tunnel, the routers decrypt the IP headers to deliver the packets to their destinations.

In transport mode, the payload of each packet is encrypted, but the original IP header is not. Intermediary routers are thus able to view the final destination of each packet — unless a separate tunneling protocol (such as GRE) is used.

What port does IPsec use?

A network port is the virtual location where data goes in a computer. Ports are how computers keep track of different processes and connections; if data goes to a certain port, the computer’s operating system knows which process it belongs to. IPsec usually uses port 500.

How does IPsec impact MSS and MTU?

MSS and MTU are two measurements of packet size. Packets can only reach a certain size (measured in bytes) before computers, routers, and switches cannot handle them. MSS measures the size of each packet’s payload, while MTU measures the entire packet, including headers. Packets that exceed a network’s MTU may be fragmented, meaning broken up into smaller packets and then reassembled. Packets that exceed the MSS are simply dropped.

IPsec protocols add several headers and trailers to packets, all of which take up several bytes. For networks that use IPsec, either the MSS and MTU have to be adjusted accordingly, or packets will be fragmented and slightly delayed. Usually, the MTU for a network is 1,500 bytes. A normal IP header is 20 bytes long, and a TCP header is also 20 bytes long, meaning each packet can contain 1,460 bytes of payload. However, IPsec adds an Authentication Header, an ESP header, and associated trailers. These add 50-60 bytes to a packet, or more.

IPsec may be used in three different security domains: virtual private networks, application-level security and routing security. At this time, IPsec is predominantly used in VPNs. When used in application-level security or routing security, IPsec is not a complete solution and must be coupled with other security measures to be effective, hindering its deployment in these domains.

IPsec operation

IPsec has two modes of operation, transport mode and tunnel mode. When operating in transport mode, the source and destination hosts must directly perform all cryptographic operations. Encrypted data is sent through a single tunnel that is created with L2TP (Layer 2 Tunneling Protocol). Data (ciphertext) is created by the source host and retrieved by the destination host. This mode of operation establishes end-to-end security.

When operating in tunnel mode, special gateways perform cryptographic processing in addition to the source and destination hosts. Here, many tunnels are created in series between gateways, establishing gateway-to-gateway security [6]. When using either of these modes, it’s important to provide all gateways with the ability to verify that a packet is real and to authenticate the packet at both ends. Any invalid packets must be dropped .

Two types of data packet encodings (DPE) are required in IPsec. These are the authentication header (AH) and the encapsulating security payload (ESP) DPEs. These encodings offer network-level security for the data [4]. The AH provides authenticity and integrity of the packet. The authentication is made available through keyed hash functions, also known as MACs (message authentication codes). This header also prohibits illegal modification and has the option of providing anti replay security. The AH can establish security between multiple hosts, multiple gateways, or multiple hosts and gateways, all implementing AH [7]. The ESP header provides encryption, data encapsulation and data confidentiality. Data confidentiality is made available through symmetric keys.

During its journey through the various tunnels and gateways, additional headers are added to the packet. On each pass through a gateway, a datagram is wrapped in a new header. Included in this header is the security parameter index (SPI). The SPI specifies the algorithms and keys that were used by the last system to view the packet. The payload is also protected in this system because any change or error in the data will be detected, causing the receiving party to drop the packet. The headers are applied at the beginning of each tunnel and then verified and removed at the end of each tunnel. This method prevents the buildup of unnecessary overhead.

An important part of IPsec is the security association (SA). The SA uses the SPI number that is carried in the AH and ESP to indicate which SA was used for the packet. An IP destination address is also included to indicate the endpoint: This may be a firewall, router or end user. A Security Association Database (SAD) is used to store all SAs that are used. A security policy is used by the SAD to indicate what the router should do with the packet. Three examples include dropping the packet altogether, dropping only the SA, or substituting a different SA. All of the security policies in use are stored in a security policy database.

Problems with IPsec

In some cases, direct end-to-end communication (i.e., transport mode) isn’t possible. The following is a simple example in which H1 and H2 are two hosts on one direct tunnel, with H1 using a firewall referred to as FW1.

“In a large distributed system or inter-domain environment, the diversified regional security policy enforcement can create significant problems for end-to-end communication. In the above example, suppose FW1 needs to examine traffic content for the purpose of intrusion detection and a policy is set up at FW1 to deny all encrypted traffic to enforce its content examination requirement. Yet, H1 and H2 build a direct tunnel without awareness of the existence of the firewall and its policy rules. Therefore, all the traffic will be dropped by FW1. The scenario shows that each policy satisfies its corresponding requirement while all policies together can cause conflicts” .

One of the biggest drawbacks of IPsec is its complexity. While IPsec’s flexibility has contributed to its popularity, it also leads to confusion and has led security experts to state that “IPsec contains too many options and too much flexibility” [8]. Much of IPsec’s flexibility and complexity may be attributed to the fact that IPsec was developed via a committee process. Due to the political nature of committees, additional features, options, and flexibility are often added to standards to satisfy various factions of the standardization body [8]. This process stands in stark contrast to the standardization process used in the development of the Advanced Encryption Standard (AES), the replacement for the Data Encryption Standard, which expired in 1998:

“It is instructive to compare this to the approach taken by NIST [the National Institute of Standards and Technology] for the development of AES. Instead of a committee, NIST organized a contest. Several small groups each created their own proposal, and the process is limited to picking one of them. At the time of writing there has been one stage of elimination, and any one of the five remaining candidates will make a much better standard than any committee could ever have made” .

Moreover, much of the documentation for IPsec is complex and confusing. No overview or introduction is provided, and nowhere are the goals of IPsec identified. The user must assemble the pieces and try to make sense of documentation that can be described as difficult to read at best. To illustrate the frustration a user must endure, consider the ISAKMP specifications. These specifications are missing many key explanations, contain numerous errors and contradict themselves in various locations .

However, while IPsec may not be perfect, it is considered a significant improvement compared with previously available security protocols. As an example, consider the popular security system Secure Sockets Layer. While SSL is widely deployed in various applications, it is inherently limited in that it is used on the transport/application layer, requiring modifications to any application that wants to include the ability to use SSL. Because IPsec is used in Layer 3, it requires modification only to the operating system rather than to the applications that employ IPsec.

Is IPsec too complex an d confusing?

IPsec incorporates all of the most commonly employed security services, including authentication, integrity, confidentiality, encryption and nonrepudiation. However, the major drawbacks to IPsec are its complexity and the confusing nature of its associated documentation. In spite of these various drawbacks, IPsec is believed by many to be one of the best security systems available. It is hoped that considerable improvement will be evidenced in future revisions of IPsec and that the problems identified with the architecture will be remedied.