Through the magic of routing, the Internet can supposedly withstand intermittent outages, congestion, and other nasties. This is because, in theory, any network has multiple paths to any other network,” sort of like highways between cities. By having multiple connections to the Internet from different providers, a network can distribute the load over all of them, choosing the best available route for each packet. This is called Multihoming, and the protocol that makes possible is BGP. The problem is that the network needs an AS number for BGP to work, and no consumer-targetted broadband connection supplies one.
Most networks that use regular broadband as their Internet connection have the following configuration. The service provider provides a DSL or Cable modem, and hooks it up to one computer to be designated the router. They generally give one or two IP addresses which may be static but are more likely dynamically allocated with DHCP. This router, which is often a computer but could also be a special-purpose appliance from Linksys or similar, sits between the network and the Internet connection. It usually runs a DHCP server on the internal network, assigning private IP addresses to all of the computers on the network. It also usually does NAT so that the computers on the Internal network can access the Internet. This works pretty well, but it's all a bit of a hack in order to get around the problem of IPv4 address exhaustion. With NAT, the entire internal network hides behind the router, so while a remote server thinks it's dealing with the router, the router knows to pass communication along to the correct internal computer.
When BGP is used to route over multiple connections, each packet can go on along a different route. However, since we're using regular consumer-grade Internet connections, each connection will have a different a IP address, and so as far as the rest of the Internet is concerned, each of them are completely unrelated. They might as well be hooked to completely separate networks. There is no way for a remote host to know that if one packet comes from one IP, and the next packet comes from a different IP, that they are both part of the same session and should be reassembled as such.
There is a way to get similar functionality by using multiple broadband connections without an AS number or any other cooperation from the service providers. It's generally referred to as "Multirouting."
Since it's fundamentally impossible to route each packet individually when NAT is involved, the best that we can do is to route each session individually. This way, we ensure that all packets from a given session use the same connection. The drawback is that we are restricted to coarser control over how data is routed.
I've successfully made this work using 4 different connections, including DSL, cable, and even a fibre connection that had an extra NAT layer upstream. This method is extremely flexible, but can be difficult to get set up correctly. Following is a contrived example using two connections to illustrate the method.
We're going to assume we've got the following interfaces set up in linux.
|Interface||Connected To||IP Address||Subnet Mask||Gateway|
We'll assume you've got all of your network interfaces configured and working. The command
ifconfig -a should list all of them, and they should all have an IP address. You should be able to ping the gateway of each of your Internet connections (
126.96.36.199 in our example), as well as any other machine on your internal network (
We'll also assume you've got NAT (aka IP Masquerading) working on each of the external interfaces (
At this point, your default gateway should be the gateway of one of your internet connections. From your router, you should be able to access the Internet, and computers on your internal network should be able to do the same. To make sure both connections are working, switch the default gateway to that of the other Internet connection like this:
# default via 188.8.131.52 dev eth2
Where 184.108.40.206 is the new gateway and eth2 is the Interface connected to that Internet connection. Once again, test that your router and internal computers can access the Internet. When testing,
traceroute is your friend to see which connection is actually being used.
Each interface is going to need its own routing table in addition to the
main routing table.
Open the file
/etc/iproute2/rt_tables in a text editor and add a line to define the names of your new routing tables. You're going to add one for each Internet connection. Each line starts with a number, then a tab, and then a name. Ours looks like this:
1 dsl_internet 2 cable_internet
Both routing tables need to know about all network interfaces, including the loopback. Each routing table should have its connection's associated gateway as its default route. You can populate them by running commands like these...
# ip route add 127.0.0.0/8 dev lo table dsl_internet # ip route add 192.168.0.0/24 dev eth0 table dsl_internet # ip route add 220.127.116.11/24 dev eth1 src 18.104.22.168 table dsl_internet # ip route add 22.214.171.124/24 dev eth2 src 126.96.36.199 table dsl_internet # ip route add default via 188.8.131.52 table dsl_internet # ip route add 127.0.0.0/8 dev lo table cable_internet # ip route add 192.168.0.0/24 dev eth0 table cable_internet # ip route add 184.108.40.206/24 dev eth1 src 220.127.116.11 table cable_internet # ip route add 18.104.22.168/24 dev eth2 src 22.214.171.124 table cable_internet # ip route add default via 126.96.36.199 table cable_internet
src address for eth0 in order to permit DNAT (port forwarding).
Now that we've created two new routing tables, we have to define the rules that will make use of them. This is especially important for DNAT (port forwarding). If a request comes in on 188.8.131.52, we definitely want to make sure that the reply goes out over the same connection, for example. These rules direct those packets to the appropriate routing tables, and can be added with these commands...
# ip rule add from 184.108.40.206 table dsl_internet # ip rule add from 220.127.116.11 table cable_internet
main routing table will probably be mostly set up already. There should be an entry for each interface. To be sure, run these commands but ignore errors if they already exist.
# ip route add 192.168.0.0/24 dev eth0 # ip route add 18.104.22.168/24 dev eth1 src 22.214.171.124 # ip route add 126.96.36.199/24 dev eth2 src 188.8.131.52
You can also use
ip route show all to see the entire
main routing table.
Now, remove the previous default route from the
main routing table like so...
# ip route del default
And add the new one that will achieve the round robin effect over all gateways... (all on one line)
# ip route add default scope global nexthop via 184.108.40.206 dev eth1 weight 1 nexthop via 220.127.116.11 dev eth2 weight 1
You're pretty much done. You should be able to access the Internet from both your router computer and your internal computers. Go to icanhazip.com on all of your computers and you should see that some have one of your IPs, and some have the other. On successive requests, you can expect it to flip flop between the two.
For your internal network, you'll likely want to run your own caching DNS server. All of your computers will query it, and it will query one or more of your ISPs' DNS servers, caching the result.
However, when your DNS server contacts one your ISPs' DNS servers, it's going to have to go through the router. You're going to want to make sure that when it contacts an ISP's DNS server, it uses that ISP's connection to do so. You'll need to set up rules to ensure that specific types of packets go over a specific connection.
Let's say the DNS server from our cable Internet provider is is
18.104.22.168. We want any packet destined for that IP to use our cable connection and not our DSL connection. This rule does just that...
# ip rule add to 22.214.171.124 table cable_internet
It says that any packet that is going to
126.96.36.199 should use the cable_internet routing table instead of the default.
Similarly, if the DNS server from our DSL provider is
# ip rule add to 188.8.131.52 table dsl_internet
Now we can tell our DNS server to forward requests to either DNS server, and our router will ensure that those requests will go out over the correct connection.
You can create more rules for each IP that you want to force down a certain connection. Some examples:
Now that you have this set up, you probably want it to work even when one of your connections goes down. Right now, if one of the connections goes down, half of the requests will fail. I wrote a script that pings the gateways every minute or so, and reconfigures the routing tables to take out unresponsive gateways until they come back. My script is ugly so I'll leave it to to write your own.