Reliable IP over multiple unrealiable IP links with Linux and Strongswan
Some time ago I wrote about how to make multiple unreliable Internet connections function like single reliable Internet connection. That was done using OpenVPN and some iptables trickery. This time we're going to do same without OpenVPN and without iptables.
We're doing layer after layer and thus reducing MTU size quite a lot. Assuming your Internet connection can handle full 1500 byte packets setup below results in 1324 byte maximum packet so overhead is 176 bytes per packet. However, be aware of MTU and GRETAP related kernel bug that has not yet been fixed. Until there's solution to that bug only using setup below in L3 routed mode makes sense. To do that skip steps 5 and 6 and set Strongswan in LAN2LAN mode on step 4.
If you're doing this over Intranet then configuration can be simplified. You can skip step 1 (Outer IPsec) and use null-crypto for step 4 (Inner IPsec). If L2 connection between end-points is not required steps 5 and 6 (Inner GRETAP) can be skipped and step 4 (Inner IPsec) be configured in lan-to-lan mode.
You can use same logic for creating load-balanced multilink VPN as well. Instead of using broadcast bonding (which duplicates packets) use loadbalance bonding. If your uplinks have different bandwidth, say 10Mbit/s and 20Mbit/s, you should be able to workaround it by creating one GRETAP over 10Mbit/s link and two GRETAP tunnels over 20Mbit/s link. Doing round-robin loadbalancing over resulting three GRETAP interfaces would then provide single 30Mbit/s connection. If you try this let me know, haven't tried it myself.
We're doing layer after layer and thus reducing MTU size quite a lot. Assuming your Internet connection can handle full 1500 byte packets setup below results in 1324 byte maximum packet so overhead is 176 bytes per packet. However, be aware of MTU and GRETAP related kernel bug that has not yet been fixed. Until there's solution to that bug only using setup below in L3 routed mode makes sense. To do that skip steps 5 and 6 and set Strongswan in LAN2LAN mode on step 4.
- Establish Strongswan IPsec MOBIKE null-crypto transport mode connection over each available Internet uplink (Outer IPsec)
- There's no need for encryption as we transport only IPsec encrypted traffic over these connections
- Create GRETAP tunnel over each of these Strongswan protected connections (Outer GRETAPs)
- Create bond interface in broadcast mode using all GRETAP tunnels
- Traffic will be sent over all GRETAP tunnels
- Establish Strongswan IPsec AES128-crypto transport mode connection over bond interface (Inner IPsec)
- This will take care of encrypting traffic and most also dropping duplicate packets received
- Create GRETAP tunnel over inner IPsec connection (Inner GRETAP)
- Bridge inner GRETAP with wired network interface
If you're doing this over Intranet then configuration can be simplified. You can skip step 1 (Outer IPsec) and use null-crypto for step 4 (Inner IPsec). If L2 connection between end-points is not required steps 5 and 6 (Inner GRETAP) can be skipped and step 4 (Inner IPsec) be configured in lan-to-lan mode.
You can use same logic for creating load-balanced multilink VPN as well. Instead of using broadcast bonding (which duplicates packets) use loadbalance bonding. If your uplinks have different bandwidth, say 10Mbit/s and 20Mbit/s, you should be able to workaround it by creating one GRETAP over 10Mbit/s link and two GRETAP tunnels over 20Mbit/s link. Doing round-robin loadbalancing over resulting three GRETAP interfaces would then provide single 30Mbit/s connection. If you try this let me know, haven't tried it myself.
# Host A # Setup interfaces ip addr add 172.31.1.1/24 dev eth1 ip addr add 172.31.2.1/24 dev eth2 ip addr add 172.31.3.1/24 dev eth3 ip addr add 172.31.4.1/24 dev eth4 ip link set dev eth1 up ip link set dev eth2 up ip link set dev eth3 up ip link set dev eth4 up # Outer IPsec without encryption # /etc/ipsec.conf cat >/etc/ipsec.conf <<__EOF__ config setup plutostart=no conn %default ikelifetime=60m keylife=20m rekeymargin=3m keyingtries=1 authby=secret mobike=yes keyexchange=ikev2 conn outer-ipsec-1 ike=aes128-sha1-modp2048! esp=null-sha1! left=172.31.1.1 leftid=@HostA-outer-1 leftfirewall=yes right=172.31.1.2 rightid=@HostB-outer-1 type=transport auto=start conn outer-ipsec-2 ike=aes128-sha1-modp2048! esp=null-sha1! left=172.31.2.1 leftid=@HostA-outer-2 leftfirewall=yes right=172.31.2.2 rightid=@HostB-outer-2 type=transport auto=start conn outer-ipsec-3 ike=aes128-sha1-modp2048! esp=null-sha1! left=172.31.3.1 leftid=@HostA-outer-3 leftfirewall=yes right=172.31.3.2 rightid=@HostB-outer-3 type=transport auto=start conn outer-ipsec-4 ike=aes128-sha1-modp2048! esp=null-sha1! left=172.31.4.1 leftid=@HostA-outer-4 leftfirewall=yes right=172.31.4.2 rightid=@HostB-outer-4 type=transport auto=start __EOF__ # /etc/ipsec.secrets cat >/etc/ipsec.secrets <<__EOF__ @HostA-outer-1 @HostB-outer-1 : PSK "NoMoreSecrets-outer-1" @HostA-outer-2 @HostB-outer-2 : PSK "NoMoreSecrets-outer-2" @HostA-outer-3 @HostB-outer-3 : PSK "NoMoreSecrets-outer-3" @HostA-outer-4 @HostB-outer-4 : PSK "NoMoreSecrets-outer-4" __EOF__ # Outer GRETAP tunnels ip link add gretap1 type gretap local 172.31.1.1 remote 172.31.1.2 key 0xbabe0001 ip link add gretap2 type gretap local 172.31.2.1 remote 172.31.2.2 key 0xbabe0002 ip link add gretap3 type gretap local 172.31.3.1 remote 172.31.3.2 key 0xbabe0003 ip link add gretap4 type gretap local 172.31.4.1 remote 172.31.4.2 key 0xbabe0004 # Outer bonding using broadcast mode modprobe bonding mode=3 ip link set dev bond0 addr 00:0c:10:20:20:01 ip link set dev bond0 up ifenslave bond0 gretap1 gretap2 gretap3 gretap4 # Packet duplicating interface used for IPsec ip addr add 10.20.20.1/24 dev bond0 # Inner IPsec with encryption # /etc/ipsec.conf cat >>/etc/ipsec.conf <<__EOF__ conn inner-ipsec left=10.20.20.1 leftid=@HostA-inner leftfirewall=yes right=10.20.20.2 rightid=@HostB-inner type=transport auto=start __EOF__ # /etc/ipsec.secrets cat >>/etc/ipsec.secrets <<__EOF__ @HostA-inner @HostB-inner : PSK "NoMoreSecrets-inner" __EOF__ # Restart IPsec /etc/init.d/ipsec restart # Inner GRETAP tunnel ip link add gretap0 type gretap local 10.20.20.1 remote 10.20.20.2 key 0xbabe0000 ip link set dev gretap0 addr 00:0c:10:10:10:01 # # Bridging gretap tunnel with physical ethernet interface ip link set dev gretap0 up ip link set dev eth5 up brctl addbr br0 brctl addif br0 gretap0 brctl addif br0 eth5 ip addr add 10.10.10.1/24 dev br0 ip link set br0 up # Done! # Host B # Setup interfaces ip addr add 172.31.1.2/24 dev eth1 ip addr add 172.31.2.2/24 dev eth2 ip addr add 172.31.3.2/24 dev eth3 ip addr add 172.31.4.2/24 dev eth4 ip link set dev eth1 up ip link set dev eth2 up ip link set dev eth3 up ip link set dev eth4 up # Outer IPsec without encryption # /etc/ipsec.conf cat >/etc/ipsec.conf <<__EOF__ config setup plutostart=no conn %default ikelifetime=60m keylife=20m rekeymargin=3m keyingtries=1 authby=secret mobike=yes keyexchange=ikev2 conn outer-ipsec-1 ike=aes128-sha1-modp2048! esp=null-sha1! left=172.31.1.2 leftid=@HostB-outer-1 leftfirewall=yes right=172.31.1.1 rightid=@HostA-outer-1 type=transport auto=start conn outer-ipsec-2 ike=aes128-sha1-modp2048! esp=null-sha1! left=172.31.2.2 leftid=@HostB-outer-2 leftfirewall=yes right=172.31.2.1 rightid=@HostA-outer-2 type=transport auto=start conn outer-ipsec-3 ike=aes128-sha1-modp2048! esp=null-sha1! left=172.31.3.2 leftid=@HostB-outer-3 leftfirewall=yes right=172.31.3.1 rightid=@HostA-outer-3 type=transport auto=start conn outer-ipsec-4 ike=aes128-sha1-modp2048! esp=null-sha1! left=172.31.4.2 leftid=@HostB-outer-4 leftfirewall=yes right=172.31.4.1 rightid=@HostA-outer-4 type=transport auto=start __EOF__ # /etc/ipsec.secrets cat >/etc/ipsec.secrets <<__EOF__ @HostB-outer-1 @HostA-outer-1 : PSK "NoMoreSecrets-outer-1" @HostB-outer-2 @HostA-outer-2 : PSK "NoMoreSecrets-outer-2" @HostB-outer-3 @HostA-outer-3 : PSK "NoMoreSecrets-outer-3" @HostB-outer-4 @HostA-outer-4 : PSK "NoMoreSecrets-outer-4" __EOF__ # GRETAP tunnels ip link add gretap1 type gretap local 172.31.1.2 remote 172.31.1.1 key 0xbabe0001 ip link add gretap2 type gretap local 172.31.2.2 remote 172.31.2.1 key 0xbabe0002 ip link add gretap3 type gretap local 172.31.3.2 remote 172.31.3.1 key 0xbabe0003 ip link add gretap4 type gretap local 172.31.4.2 remote 172.31.4.1 key 0xbabe0004 # Bonding using broadcast mode modprobe bonding mode=3 ip link set dev bond0 addr 00:0c:10:20:20:02 ip link set dev bond0 up ifenslave bond0 gretap1 gretap2 gretap3 gretap4 # Packet duplicating interface used for IPsec ip addr add 10.20.20.2/24 dev bond0 # Inner IPsec with encryption # /etc/ipsec.conf cat >>/etc/ipsec.conf <<__EOF__ conn inner-ipsec left=10.20.20.2 leftid=@HostB-inner leftfirewall=yes right=10.20.20.1 rightid=@HostA-inner type=transport auto=start __EOF__ # /etc/ipsec.secrets cat >>/etc/ipsec.secrets <<__EOF__ @HostB-inner @HostA-inner : PSK "NoMoreSecrets-inner" __EOF__ # Restart IPsec /etc/init.d/ipsec restart # Inner GRETAP tunnel ip link add gretap0 type gretap local 10.20.20.2 remote 10.20.20.1 key 0xbabe0000 ip link set dev gretap0 addr 00:0c:10:10:10:02 # # Bridging gretap tunnel with physical ethernet interface ip link set dev gretap0 up ip link set dev eth5 up brctl addbr br0 brctl addif br0 gretap0 brctl addif br0 eth5 ip addr add 10.10.10.2/24 dev br0 ip link set br0 up # Done!
Imagine how useful this article could have been to someone if you actually took the time to explain things rather than just posting a wall of code.
ReplyDeleteHi, since you seem to know IPsec/*swans, can you advice where to find proper understandable documentation about eg. libreswan implementation?
ReplyDeleteHi, the tunnel made with the approach above seems not to be accesseseble for me. Would you advice something? So, I use since one a year very similar installation but with openvpn. It is based on openwrt-router with two umts connections. Bonding with round robin increases the speed for clients dramaticaly. The only one problem is the openvpn performance on the router. So, i decided to move from openvpn user-space approach to ipsec built in kernel one. I placed stronswan on a vServer (with ubuntu trusty). It should be a gateway for clients behined the router. I made an Ipsec tunnel between the router and vServer as above. For test purpose I used only one gretap interface without bondind. I can ping both sides of the tunnel with there ip-addresses. But I cannot just make ssh-connection over the tunnel. I cannot also ping from the router any other ip on the vServer, no in the internet. I tried allready to nat gretap interfaces similar way as for openvpn. Nothing helps. Do you have any ideas?
ReplyDeleteThank you in advance.
I've got best results so far by combining MPTCP patched kernel with SoftEther VPN in TCP mode.
DeleteIndividual links (3 * LTE, 1 * ADSL) total to about 120Mbit/s and with MPTCP I get around 110Mbit/s. Running OpenVPN over TCP uses all links thanks to MPTCP, but performance is 30Mbit/s and no more than 60Mbit/s even after tweaking settings. With SoftEther I get over 90Mbit/s.
MPTCP, multiple LTE dongles, SoftEther config etc. is bit tricky to configure - especially when trying to combine them all together. I'll do post with step-by-step instructions to follow later when I have things bit more polished.
Dear asiantuntijakaveri,
ReplyDeletei am woking on IPsec with MPTCP and my use case is like
VPN tunnel
Laptop(MPTPC with multiple interfaces ips) <----------------> Test server(MPTCP)
Please let me know the complete configutation at both the side , as i am using strongswan fwk
sekhar.guruprasad@gmail.com
Thanks in advance -