Friday, December 30, 2011

(Ab)using OpenVPN to make two unrealiable Internet connections act asone reliable connection (Failure #2)

We mirror our hard disks, we mirror RAM on servers. Why not mirror network traffic as well? 



Next I tried to send packets back to self, mark them and nat with different outgoing addresses. Like previous attempt with tc even if this had worked it's horrible hack and there's still major problem of how to handle packets on server side. Especially how to duplicate return packets when client side IP addresses are dynamic.

2. Using iptables with ipip-tunnel over loopback interface

Start by allowing "send to self" i.e. don't drop packets with our own IP as source. Add these lines to /etc/sysctl.conf
net.ipv4.ip_forward=1
net.ipv6.conf.all.disable_ipv6=1
net.ipv4.conf.all.log_martians=1
net.ipv4.conf.default.rp_filter=0
net.ipv4.conf.all.rp_filter=0
net.ipv4.conf.default.accept_local=1
net.ipv4.conf.all.accept_local=1

Create un-numbered IPIP tunnel via loopback interface. You can create more than one as long as local and remote IP addresses are unique. Also adds route to 172.31.31.31/32 via ipip10, our outgoing IPIP dummy interface.
ip tunnel add ipip10 mode gre local 127.10.0.10 remote 127.10.0.11 ttl 255
ip link set ipip10 up
ip tunnel add ipip11 mode gre local 127.10.0.11 remote 127.10.0.10 ttl 255
ip link set ipip11 up
ip route add 172.31.31.31/32 dev ipip10
ip route flush cache

- Create MAKEDUPE iptables chain. Logs packets before and after processing.
- TOS flag is abused as "mark". This was part of my troubleshooting procedure but it turned out to fail exactly same way with "-j MARK --set-mark" and "-j TOS --set-tos".
- In short this sends copy of outgoing packets without TOS 0x08 to IP 172.31.31.31
- Since 172.31.31.31 is looped back to us we receive same packet back for further processing.
# MAKEDUPE chain, logs packet before, during and after processing
# tos flag is used to differentiate original and cloned packet
# copy of packet with tos 0x08 sent to 172.31.31.31 which is then received back for processing from ipip11
iptables -t mangle -N MAKEDUPE
iptables -t mangle -F MAKEDUPE
iptables -t mangle -A MAKEDUPE -j LOG --log-prefix 'MAKEDUPE Input: ' --log-level info
iptables -t mangle -A MAKEDUPE -j TOS --set-tos 0x08
iptables -t mangle -A MAKEDUPE -j LOG --log-prefix 'MAKEDUPE Clone: ' --log-level info
iptables -t mangle -A MAKEDUPE -j TEE --gateway 172.31.31.31
iptables -t mangle -A MAKEDUPE -j TOS --set-tos 0x10
iptables -t mangle -A MAKEDUPE -j LOG --log-prefix 'MAKEDUPE Output: ' --log-level info
# Send outgoing packets to MAKEDUPE chain unless packet is clone with tos 0x08
iptables -t mangle -A OUTPUT -d 172.16.0.1 -o eth0 -m tos \! --tos 0x08 -j MAKEDUPE
# NAT rules for original packet
iptables -t nat -N TOS10NAT
iptables -t nat -F TOS10NAT
iptables -t nat -A TOS10NAT -j LOG --log-prefix 'TOS10NAT: ' --log-level info
iptables -t nat -A TOS10NAT -j SNAT --to-source 192.168.0.10
# NAT rules for cloned packet
iptables -t nat -N TOS08NAT
iptables -t nat -F TOS08NAT
iptables -t nat -A TOS08NAT -j LOG --log-prefix 'TOS08NAT: ' --log-level info
iptables -t nat -A TOS08NAT -j SNAT --to-source 192.168.0.8
# Call proper NAT chain depending on tos value
iptables -A POSTROUTING -t nat -d 172.16.0.1 -m tos --tos 0x08 -g TOS08NAT
iptables -A POSTROUTING -t nat -d 172.16.0.1 -m tos --tos 0x10 -g TOS10NAT

But as I said this fails. What happens is that packets are always natted with TOS08NAT rule. Could be something with my iptables rules but result is it doesn't work as intended. See below for tcpdump capture and kernel log showing how it fails.

First ping packet is lost with 'Operation not permitted' error for some reason.
root@VPN-Client:~# ping 172.16.0.1
PING 172.16.0.1 (172.16.0.1) 56(84) bytes of data.
ping: sendmsg: Operation not permitted
^C
--- 172.16.0.1 ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3012ms

# With tcpdump we can see 192.168.0.8 source address used for both packets that have been cloned (tos 0x8, ttl 63) and original packets (tos 0x10, ttl 64). Those original packets should have 192.168.0.10 based on iptables rules.
root@VPN-Client:~# tcpdump -vv -nn -i eth0 icmp
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
14:25:55.142008 IP (tos 0x8, ttl 63, id 0, offset 0, flags [DF], proto ICMP (1), length 84)
 192.168.0.8 > 172.16.0.1: ICMP echo request, id 1340, seq 1, length 64
14:25:56.146343 IP (tos 0x8, ttl 63, id 0, offset 0, flags [DF], proto ICMP (1), length 84)
 192.168.0.8 > 172.16.0.1: ICMP echo request, id 1340, seq 2, length 64
14:25:56.146565 IP (tos 0x10, ttl 64, id 0, offset 0, flags [DF], proto ICMP (1), length 84)
 192.168.0.8 > 172.16.0.1: ICMP echo request, id 1340, seq 2, length 64
14:25:57.146178 IP (tos 0x8, ttl 63, id 0, offset 0, flags [DF], proto ICMP (1), length 84)
 192.168.0.8 > 172.16.0.1: ICMP echo request, id 1340, seq 3, length 64
14:25:57.146371 IP (tos 0x10, ttl 64, id 0, offset 0, flags [DF], proto ICMP (1), length 84)
 192.168.0.8 > 172.16.0.1: ICMP echo request, id 1340, seq 3, length 64
14:25:58.146088 IP (tos 0x8, ttl 63, id 0, offset 0, flags [DF], proto ICMP (1), length 84)
 192.168.0.8 > 172.16.0.1: ICMP echo request, id 1340, seq 4, length 64
14:25:58.146287 IP (tos 0x10, ttl 64, id 0, offset 0, flags [DF], proto ICMP (1), length 84)
 192.168.0.8 > 172.16.0.1: ICMP echo request, id 1340, seq 4, length 64

But kernel log claims packets were properly processed by TOS08NAT and TOS10NAT. Weird.
[ 116.326987] device eth0 entered promiscuous mode
[ 120.524492] MAKEDUPE Input: IN= OUT=eth0 SRC=192.168.224.21 DST=172.16.0.1 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=ICMP TYPE=8 CODE=0 ID=1340 SEQ=1
[ 120.524742] MAKEDUPE Clone: IN= OUT=eth0 SRC=192.168.224.21 DST=172.16.0.1 LEN=84 TOS=0x08 PREC=0x00 TTL=64 ID=0 DF PROTO=ICMP TYPE=8 CODE=0 ID=1340 SEQ=1
[ 120.527406] MAKEDUPE Output: IN= OUT=eth0 SRC=192.168.224.21 DST=172.16.0.1 LEN=84 TOS=0x10 PREC=0x00 TTL=64 ID=0 DF PROTO=ICMP TYPE=8 CODE=0 ID=1340 SEQ=1
[ 120.529129] TOS08NAT: IN= OUT=eth0 SRC=192.168.224.21 DST=172.16.0.1 LEN=84 TOS=0x08 PREC=0x00 TTL=63 ID=0 DF PROTO=ICMP TYPE=8 CODE=0 ID=1340 SEQ=1
[ 120.531472] TOS10NAT: IN= OUT=eth0 SRC=192.168.224.21 DST=172.16.0.1 LEN=84 TOS=0x10 PREC=0x00 TTL=64 ID=0 DF PROTO=ICMP TYPE=8 CODE=0 ID=1340 SEQ=1
[ 121.533004] MAKEDUPE Input: IN= OUT=eth0 SRC=192.168.224.21 DST=172.16.0.1 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=ICMP TYPE=8 CODE=0 ID=1340 SEQ=2
[ 121.533026] MAKEDUPE Clone: IN= OUT=eth0 SRC=192.168.224.21 DST=172.16.0.1 LEN=84 TOS=0x08 PREC=0x00 TTL=64 ID=0 DF PROTO=ICMP TYPE=8 CODE=0 ID=1340 SEQ=2
[ 121.533087] MAKEDUPE Output: IN= OUT=eth0 SRC=192.168.224.21 DST=172.16.0.1 LEN=84 TOS=0x10 PREC=0x00 TTL=64 ID=0 DF PROTO=ICMP TYPE=8 CODE=0 ID=1340 SEQ=2
[ 122.531224] MAKEDUPE Input: IN= OUT=eth0 SRC=192.168.224.21 DST=172.16.0.1 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=ICMP TYPE=8 CODE=0 ID=1340 SEQ=3
[ 122.531240] MAKEDUPE Clone: IN= OUT=eth0 SRC=192.168.224.21 DST=172.16.0.1 LEN=84 TOS=0x08 PREC=0x00 TTL=64 ID=0 DF PROTO=ICMP TYPE=8 CODE=0 ID=1340 SEQ=3
[ 122.531277] MAKEDUPE Output: IN= OUT=eth0 SRC=192.168.224.21 DST=172.16.0.1 LEN=84 TOS=0x10 PREC=0x00 TTL=64 ID=0 DF PROTO=ICMP TYPE=8 CODE=0 ID=1340 SEQ=3
[ 123.529486] MAKEDUPE Input: IN= OUT=eth0 SRC=192.168.224.21 DST=172.16.0.1 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=ICMP TYPE=8 CODE=0 ID=1340 SEQ=4
[ 123.529502] MAKEDUPE Clone: IN= OUT=eth0 SRC=192.168.224.21 DST=172.16.0.1 LEN=84 TOS=0x08 PREC=0x00 TTL=64 ID=0 DF PROTO=ICMP TYPE=8 CODE=0 ID=1340 SEQ=4
[ 123.529537] MAKEDUPE Output: IN= OUT=eth0 SRC=192.168.224.21 DST=172.16.0.1 LEN=84 TOS=0x10 PREC=0x00 TTL=64 ID=0 DF PROTO=ICMP TYPE=8 CODE=0 ID=1340 SEQ=4
[ 129.689781] device eth0 left promiscuous mode

That should be enough failures for you. In next episode you'll see how to make it actually work.

No comments:

Post a Comment

Got something to say?!