HOWTO: Multirouting with Linux
Through the magic of routing, the Internet can supposedly withstand intermittent outages, congestion, and other nasties. This is because, in theory, any network has multiple paths to any other network,” sort of like highways between cities. By having multiple connections to the Internet from different providers, a network can distribute the load over all of them, choosing the best available route for each packet. This is called Multihoming, and the protocol that makes possible is BGP. The problem is that the network needs an AS number for BGP to work, and no consumer-targetted broadband connection supplies one.
Typical small network configuration
Most networks that use regular broadband as their Internet connection have the following configuration. The service provider provides a DSL or Cable modem, and hooks it up to one computer to be designated the router. They generally give one or two IP addresses which may be static but are more likely dynamically allocated with DHCP. This router, which is often a computer but could also be a special-purpose appliance from Linksys or similar, sits between the network and the Internet connection. It usually runs a DHCP server on the internal network, assigning private IP addresses to all of the computers on the network. It also usually does NAT so that the computers on the Internal network can access the Internet. This works pretty well, but it's all a bit of a hack in order to get around the problem of IPv4 address exhaustion. With NAT, the entire internal network hides behind the router, so while a remote server thinks it's dealing with the router, the router knows to pass communication along to the correct internal computer.
Why multiple Internet connections are difficult
When BGP is used to route over multiple connections, each packet can go on along a different route. However, since we're using regular consumer-grade Internet connections, each connection will have a different a IP address, and so as far as the rest of the Internet is concerned, each of them are completely unrelated. They might as well be hooked to completely separate networks. There is no way for a remote host to know that if one packet comes from one IP, and the next packet comes from a different IP, that they are both part of the same session and should be reassembled as such.
The solution hack
There is a way to get similar functionality by using multiple broadband connections without an AS number or any other cooperation from the service providers. It's generally referred to as "Multirouting."
Since it's fundamentally impossible to route each packet individually when NAT is involved, the best that we can do is to route each session individually. This way, we ensure that all packets from a given session use the same connection. The drawback is that we are restricted to coarser control over how data is routed.
Pros
- Redundancy: If one connection goes down, the others can pick up the slack.
- Load balancing: Gain speed by spreading the bandwidth burden over all connections.
Cons
- Once a session is assigned to a link, it cannot be moved to another link. For example, if a user starts downloading a big file, and the link to which he is assigned fails half-way through, his session will terminate rather than being automatically moved to another link.
- There is no automatic way of distributing traffic based on the current available capacity of each link. The best we can do is round-robin.
- There is no automatic way of detecting a dead link. However, a simple script can send a ping across each link periodically and then reconfigure the routes to exclude a dead link (and re-enable it when it comes back up)
How to do it with Linux
I've successfully made this work using 4 different connections, including DSL, cable, and even a fibre connection that had an extra NAT layer upstream. This method is extremely flexible, but can be difficult to get set up correctly. Following is a contrived example using two connections to illustrate the method.
We're going to assume we've got the following interfaces set up in linux.
| Interface | Connected To | IP Address | Subnet Mask | Gateway |
|---|---|---|---|---|
| eth0 | Internal LAN | 192.168.0.1 | 255.255.255.0 | N/A |
| eth1 | DSL Modem | 142.154.64.7 | 255.255.255.0 | 142.154.64.1 |
| eth2 | Cable Modem | 24.19.12.45 | 255.255.255.0 | 24.19.12.1 |
1. Configure network interfaces
We'll assume you've got all of your network interfaces configured and working. The command ifconfig -a should list all of them, and they should all have an IP address. You should be able to ping the gateway of each of your Internet connections (142.154.64.1 and 24.19.12.1 in our example), as well as any other machine on your internal network (192.168.0.x).
2. Configure NAT
We'll also assume you've got NAT (aka IP Masquerading) working on each of the external interfaces (eth1 and eth2).
At this point, your default gateway should be the gateway of one of your internet connections. From your router, you should be able to access the Internet, and computers on your internal network should be able to do the same. To make sure both connections are working, switch the default gateway to that of the other Internet connection like this:
# default via 24.19.12.1 dev eth2
Where 24.19.12.1 is the new gateway and eth2 is the Interface connected to that Internet connection. Once again, test that your router and internal computers can access the Internet. When testing, traceroute is your friend to see which connection is actually being used.
3. Create the new routing tables
Each interface is going to need its own routing table in addition to the main routing table.
Open the file /etc/iproute2/rt_tables in a text editor and add a line to define the names of your new routing tables. You're going to add one for each Internet connection. Each line starts with a number, then a tab, and then a name. Ours looks like this:
1 dsl_internet 2 cable_internet
4. Populate the new routing tables
Both routing tables need to know about all network interfaces, including the loopback. Each routing table should have its connection's associated gateway as its default route. You can populate them by running commands like these...
# ip route add 127.0.0.0/8 dev lo table dsl_internet # ip route add 192.168.0.0/24 dev eth0 table dsl_internet # ip route add 142.154.64.0/24 dev eth1 src 142.154.64.7 table dsl_internet # ip route add 24.19.12.0/24 dev eth2 src 24.19.12.45 table dsl_internet # ip route add default via 142.154.64.1 table dsl_internet # ip route add 127.0.0.0/8 dev lo table cable_internet # ip route add 192.168.0.0/24 dev eth0 table cable_internet # ip route add 142.154.64.0/24 dev eth1 src 142.154.64.7 table cable_internet # ip route add 24.19.12.0/24 dev eth2 src 24.19.12.45 table cable_internet # ip route add default via 24.19.12.1 table cable_internet
Omit the src address for eth0 in order to permit DNAT (port forwarding).
5. Add rules
Now that we've created two new routing tables, we have to define the rules that will make use of them. This is especially important for DNAT (port forwarding). If a request comes in on 142.154.64.7, we definitely want to make sure that the reply goes out over the same connection, for example. These rules direct those packets to the appropriate routing tables, and can be added with these commands...
# ip rule add from 142.154.64.7 table dsl_internet # ip rule add from 24.19.12.45 table cable_internet
6. Set up the main routing table
The main routing table will probably be mostly set up already. There should be an entry for each interface. To be sure, run these commands but ignore errors if they already exist.
# ip route add 192.168.0.0/24 dev eth0 # ip route add 142.154.64.0/24 dev eth1 src 142.154.64.7 # ip route add 24.19.12.0/24 dev eth2 src 24.19.12.45
You can also use ip route show all to see the entire main routing table.
Now, remove the previous default route from the main routing table like so...
# ip route del default
And add the new one that will achieve the round robin effect over all gateways... (all on one line)
# ip route add default scope global nexthop via 142.154.64.1 dev eth1 weight 1 nexthop via 24.19.12.1 dev eth2 weight 1
Done. but...
You're pretty much done. You should be able to access the Internet from both your router computer and your internal computers. Go to icanhazip.com on all of your computers and you should see that some have one of your IPs, and some have the other. On successive requests, you can expect it to flip flop between the two.
7. Make DNS work
For your internal network, you'll likely want to run your own caching DNS server. All of your computers will query it, and it will query one or more of your ISPs' DNS servers, caching the result.
However, when your DNS server contacts one your ISPs' DNS servers, it's going to have to go through the router. You're going to want to make sure that when it contacts an ISP's DNS server, it uses that ISP's connection to do so. You'll need to set up rules to ensure that specific types of packets go over a specific connection.
Let's say the DNS server from our cable Internet provider is is 24.19.100.1. We want any packet destined for that IP to use our cable connection and not our DSL connection. This rule does just that...
# ip rule add to 24.19.100.1 table cable_internet
It says that any packet that is going to 24.19.100.1 should use the cable_internet routing table instead of the default.
Similarly, if the DNS server from our DSL provider is 142.154.61.100...
# ip rule add to 142.154.61.100 table dsl_internet
Now we can tell our DNS server to forward requests to either DNS server, and our router will ensure that those requests will go out over the correct connection.
8. Make more rules
You can create more rules for each IP that you want to force down a certain connection. Some examples:
- You forward mail through the SMTP server of one or both of your ISPs. If said SMPT server only relays from hosts on its network, you're going to need that ISP's connection.
- If one of your Internet connections is NAT'ed upstream, and they give you an IP of 10.x.x.x or something, you're going to want to make sure that all IPs destined for a 10.0.0.0/8 address go through that connection.
9. BONUS: Failover Script
Now that you have this set up, you probably want it to work even when one of your connections goes down. Right now, if one of the connections goes down, half of the requests will fail. I wrote a script that pings the gateways every minute or so, and reconfigures the routing tables to take out unresponsive gateways until they come back. My script is ugly so I'll leave it to to write your own.