(Ab)using OpenVPN to make two unrealiable Internet connections act asone reliable connection (Success!)
Hi, I'm Troy McClure. You might remember me from such posts as Failure #1 and Failure #2. Today I'm here to tell you about "OpenVPN inside OpenVPN".
Some time ago I was asked if it's possible to bond multiple 3G mobile wireless links from different operators together for one reliable connection. Bit later there was need to stream live video over 3G network from moving vehicle. Another age old question is how to make reliable low-latency VPN over public Internet by utilizing more than one provider. At that time I didn't have solution to any of these. Now I have something. It's still going to need testing and fine tuning, but at least something to try.
Say you got two Internet connections and both are equally bad with varying latency, bandwidth and packet loss. Perhaps you have two wireless links between buildings but due nature of unlicensed wireless – all those baby-monitors, video links, microwave ovens and other crap around – there’s some packet loss. Don’t worry, there’s way to make connections like these bit more reliable.
Proper solution would be of course implementing FEC. There’s few papers on subject, but no code to use so we resort to hacks. Ugly hacks. Hacks that send all traffic both ways thru all available links. Result is lesser probability for packet loss and whatever link was fastest transferring packet gets used. Downsides include large number out-of-order packets when fastest link loses packet. Duplicates are dropped early by OpenVPN’s replay protection feature and therefore they’re not visible to applications.
After banging my head against wall with horrible hacks I came to my senses. Since I already knew this was working fine between two computers connected using L2 network why not create L2 network over Internet with OpenVPN? Result will be OpenVPN inside OpenVPN. Some overhead and reduction of MTU size are downsides.
So far all my testing has been with virtual machines connected to each other. That doesn't exactly match real world situation where this hack might be needed. I'll do another post some day with real world results using multiple uplinks utilizing different technologies and different operators. Likely DSL, couple USB 3G dongles and maybe even adding Flash OFDM to the mix.
Notes below assume that you already have required packages installed and working L3 network between Client and Server. To keep things simple I'm not posting how to selectively route traffic over three different interfaces based on source/destination ports. There's plenty of documentation available on Internet for how to do it.
P.S. You can get similar results with Linux broadcast mode bonding. It's not good solution since you need to encrypt traffic twice and also end up with significant number of duplicate packets playing havoc with various applications.
That's it.
About parameters, 'mute-replay-warnings' is mandatory to avoid flooding logs with error message for every duplicated packet - and with this setup there is going to be no shortage of those.
You might want to tune 'replay-window' setting depending on your link speed and latency. It's currently set to accept up to 512 packets out-of-order but no packets that are more than 10 seconds late. It might be best to shorten timeout as I don't think kernel nor applications really expect packet that is normally received in 10ms to come after 10 seconds.
Since different paths can have different MTU sizes we're limiting main OpenVPN process (one that does encryption) to maximum of 1372 bytes including any OpenVPN added headers. Hopefully this is small enough to pass thru any uplinks we might have. If not reduce values of both 'fragment' and 'mssfix' by same amount. Assuming I've interpreted OpenVPN docs correctly settings above allow transferring full 1500 byte IP packet but "recommend" that clients limit packet size to 1280 bytes. Anything that will be after compression, encryption and adding headers over 1372 byte limit will be fragmented to two approximately equal sized packets. Reason for allowing fragmentation with large packets is to make OpenVPN link as transparent as possible.
If you're CPU bound or have fast uplinks disabling adaptive compression (comp-lzo) is probably good idea. Passing TOS bits ('passtos') gives no performance improvements in typical setup. However by keeping TOS bits "public" you can use them to prioritize encrypted packets per-uplink when they're leaving your router. For this just follow your favourite QoS document to mark packets when they enter your router and then apply policing to those OpenVPN connections going out over your ISP links.
To test above setup try sending some traffic between 172.31.88.1 (server) and 172.31.88.2 (client). All packets, both ways, will be sent three times but apps only see one of them.
I tried applying 5% input and 5% output packet loss on VMware Workstation settings. Result was ping going directly without OpenVPN tunneling showed 17% packet loss after 12 hours. One using three tunnels showed 0,07% packet loss. Very un-scientific test, but at least I know that it's doing what I'm expecting.
Some time ago I was asked if it's possible to bond multiple 3G mobile wireless links from different operators together for one reliable connection. Bit later there was need to stream live video over 3G network from moving vehicle. Another age old question is how to make reliable low-latency VPN over public Internet by utilizing more than one provider. At that time I didn't have solution to any of these. Now I have something. It's still going to need testing and fine tuning, but at least something to try.
Say you got two Internet connections and both are equally bad with varying latency, bandwidth and packet loss. Perhaps you have two wireless links between buildings but due nature of unlicensed wireless – all those baby-monitors, video links, microwave ovens and other crap around – there’s some packet loss. Don’t worry, there’s way to make connections like these bit more reliable.
Proper solution would be of course implementing FEC. There’s few papers on subject, but no code to use so we resort to hacks. Ugly hacks. Hacks that send all traffic both ways thru all available links. Result is lesser probability for packet loss and whatever link was fastest transferring packet gets used. Downsides include large number out-of-order packets when fastest link loses packet. Duplicates are dropped early by OpenVPN’s replay protection feature and therefore they’re not visible to applications.
After banging my head against wall with horrible hacks I came to my senses. Since I already knew this was working fine between two computers connected using L2 network why not create L2 network over Internet with OpenVPN? Result will be OpenVPN inside OpenVPN. Some overhead and reduction of MTU size are downsides.
So far all my testing has been with virtual machines connected to each other. That doesn't exactly match real world situation where this hack might be needed. I'll do another post some day with real world results using multiple uplinks utilizing different technologies and different operators. Likely DSL, couple USB 3G dongles and maybe even adding Flash OFDM to the mix.
Notes below assume that you already have required packages installed and working L3 network between Client and Server. To keep things simple I'm not posting how to selectively route traffic over three different interfaces based on source/destination ports. There's plenty of documentation available on Internet for how to do it.
P.S. You can get similar results with Linux broadcast mode bonding. It's not good solution since you need to encrypt traffic twice and also end up with significant number of duplicate packets playing havoc with various applications.
- Configure OpenVPN on server side.
# Server mkdir -p /etc/openvpn cd /etc/openvpn # 1st link is named 101 openvpn --genkey --secret openvpn101.key cat > openvpn101.conf <<__END__ port 20101 dev tap101 ifconfig 172.31.101.1 255.255.255.0 verb 4 secret openvpn101.key cipher none __END__ # 2nd link is named 102 openvpn --genkey --secret openvpn102.key cat > openvpn102.conf <<__END__ port 20102 dev tap102 ifconfig 172.31.102.1 255.255.255.0 verb 4 secret openvpn102.key cipher none __END__ # 3rd link is named 103 openvpn --genkey --secret openvpn103.key cat > openvpn103.conf <<__END__ port 20103 dev tap103 ifconfig 172.31.103.1 255.255.255.0 verb 4 secret openvpn103.key cipher none __END__ # main link is named 088 openvpn --genkey --secret openvpn088.key cat > openvpn088.conf <<__END__ local 172.31.255.1 port 20088 dev tap088 ifconfig 172.31.88.1 255.255.255.0 verb 4 secret openvpn088.key mute-replay-warnings replay-window 512 10 tun-mtu 1500 fragment 1372 mssfix 1282 passtos comp-lzo __END__
- Configure OpenVPN on client side.
# Client mkdir -p /etc/openvpn cd /etc/openvpn scp 172.16.0.1:/etc/openvpn/openvpn*.key . # 1st cat > openvpn101.conf <<__END__ remote 172.16.0.1 20101 port 20101 dev tap101 ifconfig 172.31.101.2 255.255.255.0 verb 4 secret openvpn101.key cipher none __END__ # 2nd cat > openvpn102.conf <<__END__ remote 172.16.0.1 20102 port 20102 dev tap102 ifconfig 172.31.102.2 255.255.255.0 verb 4 secret openvpn102.key cipher none __END__ # 3rd cat > openvpn103.conf <<__END__ remote 172.16.0.1 20103 port 20103 dev tap103 ifconfig 172.31.103.2 255.255.255.0 verb 4 secret openvpn103.key cipher none __END__ # main cat > openvpn088.conf <<__END__ local 172.31.255.2 remote 172.31.255.1 20088 port 20088 dev tap088 ifconfig 172.31.88.2 255.255.255.0 verb 4 secret openvpn088.key mute-replay-warnings replay-window 512 10 tun-mtu 1500 fragment 1372 mssfix 1282 passtos comp-lzo __END__
- Launch OpenVPN and add required iptables rules to server.
# SERVER cd /etc/openvpn; openvpn --config openvpn101.conf --daemon cd /etc/openvpn; openvpn --config openvpn102.conf --daemon cd /etc/openvpn; openvpn --config openvpn103.conf --daemon # Magic starts here. We need "always on" connection and dummy0 does exactly that modprobe dummy ifconfig dummy0 down ifconfig dummy0 172.31.255.1 netmask 255.255.255.255 up ip route add 172.31.255.2 dev dummy0 # Create three copies of OpenVPN packets # Would be nice to discard original but doing so sends false icmp unreachables even with -j DROP iptables -t mangle -A OUTPUT -d 172.31.255.2 -o dummy0 -j TEE --gateway 172.31.101.2 iptables -t mangle -A OUTPUT -d 172.31.255.2 -o dummy0 -j TEE --gateway 172.31.102.2 iptables -t mangle -A OUTPUT -d 172.31.255.2 -o dummy0 -j TEE --gateway 172.31.103.2 # Launch actual encrypting OpenVPN process cd /etc/openvpn; openvpn --config openvpn088.conf
- Launch OpenVPN and add required iptables rules to client.
# CLIENT cd /etc/openvpn; openvpn --config openvpn101.conf --daemon cd /etc/openvpn; openvpn --config openvpn102.conf --daemon cd /etc/openvpn; openvpn --config openvpn103.conf --daemon # Magic starts here. We need "always on" connection and dummy0 does exactly that modprobe dummy ifconfig dummy0 down ifconfig dummy0 172.31.255.2 netmask 255.255.255.255 up ip route add 172.31.255.1 dev dummy0 # Create three copies of OpenVPN packets # Would be nice to discard original but doing so sends false icmp unreachables even with -j DROP iptables -t mangle -A OUTPUT -d 172.31.255.1 -o dummy0 -j TEE --gateway 172.31.101.1 iptables -t mangle -A OUTPUT -d 172.31.255.1 -o dummy0 -j TEE --gateway 172.31.102.1 iptables -t mangle -A OUTPUT -d 172.31.255.1 -o dummy0 -j TEE --gateway 172.31.103.1 # Launch actual encrypting OpenVPN process cd /etc/openvpn; openvpn --config openvpn088.conf
That's it.
About parameters, 'mute-replay-warnings' is mandatory to avoid flooding logs with error message for every duplicated packet - and with this setup there is going to be no shortage of those.
You might want to tune 'replay-window' setting depending on your link speed and latency. It's currently set to accept up to 512 packets out-of-order but no packets that are more than 10 seconds late. It might be best to shorten timeout as I don't think kernel nor applications really expect packet that is normally received in 10ms to come after 10 seconds.
Since different paths can have different MTU sizes we're limiting main OpenVPN process (one that does encryption) to maximum of 1372 bytes including any OpenVPN added headers. Hopefully this is small enough to pass thru any uplinks we might have. If not reduce values of both 'fragment' and 'mssfix' by same amount. Assuming I've interpreted OpenVPN docs correctly settings above allow transferring full 1500 byte IP packet but "recommend" that clients limit packet size to 1280 bytes. Anything that will be after compression, encryption and adding headers over 1372 byte limit will be fragmented to two approximately equal sized packets. Reason for allowing fragmentation with large packets is to make OpenVPN link as transparent as possible.
If you're CPU bound or have fast uplinks disabling adaptive compression (comp-lzo) is probably good idea. Passing TOS bits ('passtos') gives no performance improvements in typical setup. However by keeping TOS bits "public" you can use them to prioritize encrypted packets per-uplink when they're leaving your router. For this just follow your favourite QoS document to mark packets when they enter your router and then apply policing to those OpenVPN connections going out over your ISP links.
To test above setup try sending some traffic between 172.31.88.1 (server) and 172.31.88.2 (client). All packets, both ways, will be sent three times but apps only see one of them.
I tried applying 5% input and 5% output packet loss on VMware Workstation settings. Result was ping going directly without OpenVPN tunneling showed 17% packet loss after 12 hours. One using three tunnels showed 0,07% packet loss. Very un-scientific test, but at least I know that it's doing what I'm expecting.
Comments
Post a Comment
Got something to say?!