To configure OpenWRT to two ISP's, we will need to configure at least 3 interfaces: WAN-1 for ISP-1, WAN-2 for ISP-2, and LAN. Note that WAN-1 uses WAN port, WAN-2 uses one of the 4 Ethernet Switch ports. The remaining 3 EthernetSwitch ports plus the Wi-Fi form LAN.

The Interface names for LAN is vlan0, first WAN is vlan1, and second WAN is vlan2.

The nvram setting should be:

lan_ifname=br0
lan_ifnames="vlan0 eth1"
lan_proto=static
lan_ipaddr=192.168.1.1
lan_netmask=255.255.255.0
wan_ifname=vlan1
wan2_iframe=vlan2

vlan0ports="2 3 4 5*"
vlan0hwname=et0
vlan1ports="0 5"
vlan1hwname=et0
vlan2ports="1 5"
vlan2hwname=et0

                                              /
           +-----------------+   +---------+ DSL link X / +------
           | 11.1.1.1  WAN1  +---+ Modem X +--------------+ ISP X
           |                 |   +---------+           |  +------
-----------| OpenWRT	     |                         |        Internet
  LAN      |                 |   +---------+           |  +------
192.168.1.1| 22.2.2.2  WAN2  +---+ Modem Y +--------------+ ISP Y
           +-----------------+   +---------+ DSL link Y \ +------
                                              \

After the Interface part is fixed, the next step is to configure the route(s).

We will need to download the iproute2 and iptables-saverestore package in order to adapt teh following rules:

 
Incoming connections 
 
Outgoing packets belong to the connection from X.X.X.X has the source of 11.1.1.1,
and outgoing packets of the connection from Y.Y.Y.Y has the source of 22.2.2.2.
 
Thus we need to create 2 tables - one for each of the sources: 

# ip rule add prio 1 from 11.1.1.1 lookup 1
# ip rule add prio 2 from 22.2.2.2 lookup 2

Set default routes for packets from each source: 

# ip route add table 1 to default dev vlan1

# ip route add table 2 to default dev vlan2

The above routing setup will allow the incoming traffic from DSL Link X to go back to ISP X, while the incoming traffic from DSL Link Y to go back to ISP Y. If there is a Web Server in OpenWRT, this sever can serve two IP's from two ISP's already. If we setup Round Robin DNS for this server, then it has failover and load balance capabilities.

Load balancing outgoing connections 

To take care of outboiund traffic, we will need to do multipath route:

# ip route add to default nexthop dev vlan1 nexthop dev vlan2
 
 
Acting as an internet gateway 
             Normally, this OpenWRT is setup as a pure router to let the
LAN inside to connect to both 
WAN-1 an dWAN2 with failover and load-balance.

 
 
 
 
Firstly - IP forwarding is required: 
# echo 1 > /proc/sys/net/ipv4/ip_forward
We will need CONNMARK to memorize the link we picked before.
 
Now, mark first packets based on the interface selectected by the multipath route 
# iptables -A POSTROUTING -t mangle -j MARK --set-mark 1 -m state --state NEW -o vlan1
# iptables -A POSTROUTING -t mangle -j MARK --set-mark 2 -m state --state NEW -o vlan2
# iptables -A POSTROUTING -t mangle -j CONNMARK --save-mark -m state --state NEW

For following packets, we will restore the saved mark before reaching the routing system 
# iptables -A PREROUTING -t mangle -j CONNMARK --restore-mark
Route following packets based on the restored mark 
# ip rule add fwmark 1 lookup 1
# ip rule add fwmark 2 lookup 2

 

The last step - Source-NAT them to the public IP addresses before send to the internet so the

                  returning packets know where to go back to us 
# iptables -A POSTROUTING -t nat -m mark --mark 1 -j SNAT --to-source 11.1.1.1
# iptables -A POSTROUTING -t nat -m mark --mark 2 -j SNAT --to-source 22.2.2.2

Reference
http://routeskeeper.sourceforge.net/cgi-bin/twiki/view/Routeskeeper/MultihomingWithLinux
http://wiki.openwrt.org/ConfigurableFirewall
http://wiki.openwrt.org/OpenWrtDocs/Configuration
http://snafu.freedom.org/linux2.2/iproute-notes.html#synip
http://forum.openwrt.org/viewtopic.php?id=2314