Thursday, January 19, 2012

Reliable IP over multiple unrealiable IP links with Linux and Strongswan

Some time ago I wrote about how to make multiple unreliable Internet connections function like single reliable Internet connection. That was done using OpenVPN and some iptables trickery. This time we're going to do same without OpenVPN and without iptables.



We're doing layer after layer and thus reducing MTU size quite a lot. Assuming your Internet connection can handle full 1500 byte packets setup below results in 1324 byte maximum packet so overhead is 176 bytes per packet. However, be aware of MTU and GRETAP related kernel bug that has not yet been fixed. Until there's solution to that bug only using setup below in L3 routed mode makes sense. To do that skip steps 5 and 6 and set Strongswan in LAN2LAN mode on step 4.

  1. Establish Strongswan IPsec MOBIKE null-crypto transport mode connection over each available Internet uplink (Outer IPsec)

    • There's no need for encryption as we transport only IPsec encrypted traffic over these connections

  2. Create GRETAP tunnel over each of these Strongswan protected connections (Outer GRETAPs)

  3. Create bond interface in broadcast mode using all GRETAP tunnels

    • Traffic will be sent over all GRETAP tunnels

  4. Establish Strongswan IPsec AES128-crypto transport mode connection over bond interface (Inner IPsec)

    • This will take care of encrypting traffic and most also dropping duplicate packets received

  5. Create GRETAP tunnel over inner IPsec connection (Inner GRETAP)

  6. Bridge inner GRETAP with wired network interface

If you're doing this over Intranet then configuration can be simplified. You can skip step 1 (Outer IPsec) and use null-crypto for step 4 (Inner IPsec). If L2 connection between end-points is not required steps 5 and 6 (Inner GRETAP) can be skipped and step 4 (Inner IPsec) be configured in lan-to-lan mode.

You can use same logic for creating load-balanced multilink VPN as well. Instead of using broadcast bonding (which duplicates packets) use loadbalance bonding. If your uplinks have different bandwidth, say 10Mbit/s and 20Mbit/s, you should be able to workaround it by creating one GRETAP over 10Mbit/s link and two GRETAP tunnels over 20Mbit/s link. Doing round-robin loadbalancing over resulting three GRETAP interfaces would then provide single 30Mbit/s connection. If you try this let me know, haven't tried it myself.
# Host A

# Setup interfaces
ip addr add 172.31.1.1/24 dev eth1
ip addr add 172.31.2.1/24 dev eth2
ip addr add 172.31.3.1/24 dev eth3
ip addr add 172.31.4.1/24 dev eth4
ip link set dev eth1 up
ip link set dev eth2 up
ip link set dev eth3 up
ip link set dev eth4 up

# Outer IPsec without encryption
# /etc/ipsec.conf
cat >/etc/ipsec.conf <<__EOF__
config setup
 plutostart=no
conn %default
 ikelifetime=60m
 keylife=20m
 rekeymargin=3m
 keyingtries=1
 authby=secret
 mobike=yes
 keyexchange=ikev2
conn outer-ipsec-1
 ike=aes128-sha1-modp2048!
 esp=null-sha1!
 left=172.31.1.1
 leftid=@HostA-outer-1
 leftfirewall=yes
 right=172.31.1.2
 rightid=@HostB-outer-1
 type=transport
 auto=start
conn outer-ipsec-2
 ike=aes128-sha1-modp2048!
 esp=null-sha1!
 left=172.31.2.1
 leftid=@HostA-outer-2
 leftfirewall=yes
 right=172.31.2.2
 rightid=@HostB-outer-2
 type=transport
 auto=start
conn outer-ipsec-3
 ike=aes128-sha1-modp2048!
 esp=null-sha1!
 left=172.31.3.1
 leftid=@HostA-outer-3
 leftfirewall=yes
 right=172.31.3.2
 rightid=@HostB-outer-3
 type=transport
 auto=start
conn outer-ipsec-4
 ike=aes128-sha1-modp2048!
 esp=null-sha1!
 left=172.31.4.1
 leftid=@HostA-outer-4
 leftfirewall=yes
 right=172.31.4.2
 rightid=@HostB-outer-4
 type=transport
 auto=start
__EOF__

# /etc/ipsec.secrets
cat >/etc/ipsec.secrets <<__EOF__
@HostA-outer-1 @HostB-outer-1 : PSK "NoMoreSecrets-outer-1"
@HostA-outer-2 @HostB-outer-2 : PSK "NoMoreSecrets-outer-2"
@HostA-outer-3 @HostB-outer-3 : PSK "NoMoreSecrets-outer-3"
@HostA-outer-4 @HostB-outer-4 : PSK "NoMoreSecrets-outer-4"
__EOF__

# Outer GRETAP tunnels
ip link add gretap1 type gretap local 172.31.1.1 remote 172.31.1.2 key 0xbabe0001
ip link add gretap2 type gretap local 172.31.2.1 remote 172.31.2.2 key 0xbabe0002
ip link add gretap3 type gretap local 172.31.3.1 remote 172.31.3.2 key 0xbabe0003
ip link add gretap4 type gretap local 172.31.4.1 remote 172.31.4.2 key 0xbabe0004

# Outer bonding using broadcast mode
modprobe bonding mode=3
ip link set dev bond0 addr 00:0c:10:20:20:01
ip link set dev bond0 up
ifenslave bond0 gretap1 gretap2 gretap3 gretap4

# Packet duplicating interface used for IPsec
ip addr add 10.20.20.1/24 dev bond0

# Inner IPsec with encryption
# /etc/ipsec.conf
cat >>/etc/ipsec.conf <<__EOF__
conn inner-ipsec
 left=10.20.20.1
 leftid=@HostA-inner
 leftfirewall=yes
 right=10.20.20.2
 rightid=@HostB-inner
 type=transport
 auto=start
__EOF__

# /etc/ipsec.secrets
cat >>/etc/ipsec.secrets <<__EOF__
@HostA-inner @HostB-inner : PSK "NoMoreSecrets-inner"
__EOF__

# Restart IPsec
/etc/init.d/ipsec restart

# Inner GRETAP tunnel
ip link add gretap0 type gretap local 10.20.20.1 remote 10.20.20.2 key 0xbabe0000
ip link set dev gretap0 addr 00:0c:10:10:10:01

#
# Bridging gretap tunnel with physical ethernet interface
ip link set dev gretap0 up
ip link set dev eth5 up
brctl addbr br0
brctl addif br0 gretap0
brctl addif br0 eth5
ip addr add 10.10.10.1/24 dev br0
ip link set br0 up

# Done!
# Host B

# Setup interfaces
ip addr add 172.31.1.2/24 dev eth1
ip addr add 172.31.2.2/24 dev eth2
ip addr add 172.31.3.2/24 dev eth3
ip addr add 172.31.4.2/24 dev eth4
ip link set dev eth1 up
ip link set dev eth2 up
ip link set dev eth3 up
ip link set dev eth4 up

# Outer IPsec without encryption
# /etc/ipsec.conf
cat >/etc/ipsec.conf <<__EOF__
config setup
 plutostart=no
conn %default
 ikelifetime=60m
 keylife=20m
 rekeymargin=3m
 keyingtries=1
 authby=secret
 mobike=yes
 keyexchange=ikev2
conn outer-ipsec-1
 ike=aes128-sha1-modp2048!
 esp=null-sha1!
 left=172.31.1.2
 leftid=@HostB-outer-1
 leftfirewall=yes
 right=172.31.1.1
 rightid=@HostA-outer-1
 type=transport
 auto=start
conn outer-ipsec-2
 ike=aes128-sha1-modp2048!
 esp=null-sha1!
 left=172.31.2.2
 leftid=@HostB-outer-2
 leftfirewall=yes
 right=172.31.2.1
 rightid=@HostA-outer-2
 type=transport
 auto=start
conn outer-ipsec-3
 ike=aes128-sha1-modp2048!
 esp=null-sha1!
 left=172.31.3.2
 leftid=@HostB-outer-3
 leftfirewall=yes
 right=172.31.3.1
 rightid=@HostA-outer-3
 type=transport
 auto=start
conn outer-ipsec-4
 ike=aes128-sha1-modp2048!
 esp=null-sha1!
 left=172.31.4.2
 leftid=@HostB-outer-4
 leftfirewall=yes
 right=172.31.4.1
 rightid=@HostA-outer-4
 type=transport
 auto=start
__EOF__

# /etc/ipsec.secrets
cat >/etc/ipsec.secrets <<__EOF__
@HostB-outer-1 @HostA-outer-1 : PSK "NoMoreSecrets-outer-1"
@HostB-outer-2 @HostA-outer-2 : PSK "NoMoreSecrets-outer-2"
@HostB-outer-3 @HostA-outer-3 : PSK "NoMoreSecrets-outer-3"
@HostB-outer-4 @HostA-outer-4 : PSK "NoMoreSecrets-outer-4"
__EOF__

# GRETAP tunnels
ip link add gretap1 type gretap local 172.31.1.2 remote 172.31.1.1 key 0xbabe0001
ip link add gretap2 type gretap local 172.31.2.2 remote 172.31.2.1 key 0xbabe0002
ip link add gretap3 type gretap local 172.31.3.2 remote 172.31.3.1 key 0xbabe0003
ip link add gretap4 type gretap local 172.31.4.2 remote 172.31.4.1 key 0xbabe0004

# Bonding using broadcast mode
modprobe bonding mode=3
ip link set dev bond0 addr 00:0c:10:20:20:02
ip link set dev bond0 up
ifenslave bond0 gretap1 gretap2 gretap3 gretap4

# Packet duplicating interface used for IPsec
ip addr add 10.20.20.2/24 dev bond0

# Inner IPsec with encryption
# /etc/ipsec.conf
cat >>/etc/ipsec.conf <<__EOF__
conn inner-ipsec
 left=10.20.20.2
 leftid=@HostB-inner
 leftfirewall=yes
 right=10.20.20.1
 rightid=@HostA-inner
 type=transport
 auto=start
__EOF__

# /etc/ipsec.secrets
cat >>/etc/ipsec.secrets <<__EOF__
@HostB-inner @HostA-inner : PSK "NoMoreSecrets-inner"
__EOF__

# Restart IPsec
/etc/init.d/ipsec restart

# Inner GRETAP tunnel
ip link add gretap0 type gretap local 10.20.20.2 remote 10.20.20.1 key 0xbabe0000
ip link set dev gretap0 addr 00:0c:10:10:10:02

#
# Bridging gretap tunnel with physical ethernet interface
ip link set dev gretap0 up
ip link set dev eth5 up
brctl addbr br0
brctl addif br0 gretap0
brctl addif br0 eth5
ip addr add 10.10.10.2/24 dev br0
ip link set br0 up

# Done!

5 comments:

  1. Imagine how useful this article could have been to someone if you actually took the time to explain things rather than just posting a wall of code.

    ReplyDelete
  2. Hi, since you seem to know IPsec/*swans, can you advice where to find proper understandable documentation about eg. libreswan implementation?

    ReplyDelete
  3. Hi, the tunnel made with the approach above seems not to be accesseseble for me. Would you advice something? So, I use since one a year very similar installation but with openvpn. It is based on openwrt-router with two umts connections. Bonding with round robin increases the speed for clients dramaticaly. The only one problem is the openvpn performance on the router. So, i decided to move from openvpn user-space approach to ipsec built in kernel one. I placed stronswan on a vServer (with ubuntu trusty). It should be a gateway for clients behined the router. I made an Ipsec tunnel between the router and vServer as above. For test purpose I used only one gretap interface without bondind. I can ping both sides of the tunnel with there ip-addresses. But I cannot just make ssh-connection over the tunnel. I cannot also ping from the router any other ip on the vServer, no in the internet. I tried allready to nat gretap interfaces similar way as for openvpn. Nothing helps. Do you have any ideas?
    Thank you in advance.

    ReplyDelete
    Replies
    1. I've got best results so far by combining MPTCP patched kernel with SoftEther VPN in TCP mode.

      Individual links (3 * LTE, 1 * ADSL) total to about 120Mbit/s and with MPTCP I get around 110Mbit/s. Running OpenVPN over TCP uses all links thanks to MPTCP, but performance is 30Mbit/s and no more than 60Mbit/s even after tweaking settings. With SoftEther I get over 90Mbit/s.

      MPTCP, multiple LTE dongles, SoftEther config etc. is bit tricky to configure - especially when trying to combine them all together. I'll do post with step-by-step instructions to follow later when I have things bit more polished.

      Delete
  4. Dear asiantuntijakaveri,

    i am woking on IPsec with MPTCP and my use case is like

    VPN tunnel
    Laptop(MPTPC with multiple interfaces ips) <----------------> Test server(MPTCP)

    Please let me know the complete configutation at both the side , as i am using strongswan fwk

    sekhar.guruprasad@gmail.com

    Thanks in advance -

    ReplyDelete

Got something to say?!