双线路问题

开始

还是去年的这个时候，李硕问我在FreeBSD上有没有网吧（电信、网通）双线路的解决办法，共同研究。我说有，但是上网一找却没找到。后来李硕再没提此事，就忘了。

后来，韩冬媳妇问我会不会做网吧双线，说可以用来联系电信的业务。我说可以试试看。再后来，结婚，去哈尔滨学习水手业务，从哈尔滨回来就回家过年，然后找工作，上班。这件事一直没有结果。

直到上个月，给吉林教育杂志社做网络共享连接，一时兴起用 m0n0Wall 做了，又想起这件事来。

离开方正维修站后。暂时没有好的想法和打算，于是趁机研究了一下。

实践

起初，在网上找相关的文章，大多是用专用路由器解决。除了那些专用设备以外我能接触上的只有：Route OS 、 m0n0Wall 、Iptables 、ipfw、ipf 和 pf 。我一个一个的试。其实这些东西都是我所不了解的，下载找资料安装测试就花去了很多时间。后来我决定从 pf 开始着手，因为这个我手上的资料最多，FreeBSD 是我相对来说最熟悉的。

以前用过 ipfw 、 ipfilter 之后就没用过 pf ，主要是和他们老是弄混。一个功能的实现方式不一样，所以学着很别扭。用过了以后才知道 pf 真的是很好的一个防火墙系统，我很喜欢。

我给自己搭建了以下的环境：


pc1 ： ---------------------| | pc2 : --------------------------------- || modem：

FreeBSD 4.11 + ipfilter ----| | FreeBSD 6.2 + pf

rl0 192.168.1.254 ---------|-|----------------------------------------> (to modem)

rl1 172.16.11.1/24 <-------|-|-> fxp0 172.16.11.2/24 (default 172.16.11.1)

rl2 172.17.12.1/24 <------- |-|-> fxp1 172.16.12.2/24


pc3 :Windowx 2003 -------|
192.168.0.2/24 -default- <-|---> rl0 192.168.0.1/24
Windows 2003

pc1 设置：

自动 PPPoE 拨号
ipfilter 共享上网，一切从简。
pass in quick all
pass out quick all
map tun0 172.16.0.0/16 -> 0/32 portmap tcp/udp 10000:40000
map tun0 172.16.0.0/16 -> 0/32

中间忘记了启用ip转发，上不了网，浪费了一些时间，晕一下。

pc2 设置：

1.表

table 《表名》 persist file "/etc/telecom_ip"

2.规则
ext_if1 = "fxp0"
ext_gw2 = "172.16.11.1"
ext_if2 = "fxp1"
ext_gw2 = "172.16.12.1"
int_if = "rl0:
int_net = "192.168.0.0/24"

table perstst files "/etc/telecom_ip"

下面是手册中的说明，我按照他给出的例子做了修改。

#下面的例子通过2条到因特网的连接平衡输出流量：
#pass in on $int_if route-to {($ext_if1 $ext_gw2), ($ext_if2 $ext_gw2)} round-robin from $int_net to any keep state

# route-to 选项用来在收到流量的内部接口上指定平衡的流量经过各自的网关到输出的网络接口。
# 注意route-to 选项必须在每个需要均衡的过滤规则上出现。返回的数据包会路由到它们出去时的外部接口（这是由ISP做的），然后正常路由回内部网络。
# 要保证带有属于$ext_if1源地址的数据包总是路由到$ext_gw1（$ext_if2 和 $ext_gw2也是同样的），下面2行必须包括在规则集中：
pass out on $ext_if1 route-to ($ext_if2 $ext_gw2) from $ext_if2 to any
pass out on $ext_if2 route-to ($ext_if2 $ext_gw2) from $ext_if1 to any

# 最后，NAT也可以使用在输出接口中：
nat on $ext_if2 from $int_net to -> $ext_if2
nat on $ext_if1 from $int_net to any -> $ext_if1

实验到晚上，我发现，问题的关键在于静态路由表：不添加静态路由表的话，走另外一条线路的访问不会回来。但是做负载均衡的可以。目前来看，只要加了静态路由表，做一下两个方向的NAT 规则就可已各走各的路。不加静态路由表，怎么做都没有用。明天加一台机器再测试。

下面是第二天（12:56 2007-6-6）做的实验。

以下的这个规则是从手册上修改过来的，他的确可以实现负载均衡。

# cat pf.2
lan_net = "192.168.0.0/24"
int_if = "rl0"
ext_if1 = "fxp0"
ext_if2 = "fxp1"
ext_gw1 = "172.16.11.1"
ext_gw2 = "172.16.12.1"

table persist file "/etc/telecom_ip"


# nat outgoing connections on each internet interface
nat on $ext_if1 from $lan_net to any -> ($ext_if1)
nat on $ext_if2 from $lan_net to any -> ($ext_if2)

# default deny
block in from any to any
block out from any to any

# pass all outgoing packets on internal interface
pass out on $int_if from any to $lan_net

# pass in quick any packets destined for the gateway itself
pass in quick on $int_if from $lan_net to $int_if

# load balance outgoing tcp traffic from internal network.
pass in on $int_if route-to \
{ ($ext_if1 $ext_gw1), ($ext_if2 $ext_gw2) } round-robin \
proto tcp from $lan_net to any flags S/SA modulate state

# load balance outgoing udp and icmp traffic from internal network
pass in on $int_if route-to \
{ ($ext_if1 $ext_gw1), ($ext_if2 $ext_gw2) } round-robin \
proto { udp, icmp } from $lan_net to any keep state

# general "pass out" rules for external interfaces
pass out on $ext_if1 proto tcp from any to any flags S/SA modulate state
pass out on $ext_if1 proto { udp, icmp } from any to any keep state
pass out on $ext_if2 proto tcp from any to any flags S/SA modulate state
pass out on $ext_if2 proto { udp, icmp } from any to any keep state

# route packets from any IPs on $ext_if1 to $ext_gw1 and the same for
# $ext_if2 and $ext_gw2
pass out on $ext_if1 route-to ($ext_if2 $ext_gw2) from $ext_if2 to any
pass out on $ext_if2 route-to ($ext_if1 $ext_gw1) from $ext_if1 to any

既然双线路出口都有数据通过，似乎加不加默认网关已经没什么作用了。于是我删除了默认网关。没错，在pc2 上已经不能 nslookup 了，但是在 2003 上可以，同样可以下载文件，同样是负载均衡状态，两条出口线路都有数据经过。

最后我发现：问题出现在问题出在局域网入口上：

# load balance outgoing tcp traffic from internal network.
pass in on $int_if route-to \
{ ($ext_if1 $ext_gw1), ($ext_if2 $ext_gw2) } round-robin \
proto tcp from $lan_net to any flags S/SA modulate state

# load balance outgoing udp and icmp traffic from internal network
pass in on $int_if route-to \
{ ($ext_if1 $ext_gw1), ($ext_if2 $ext_gw2) } round-robin \
proto { udp, icmp } from $lan_net to any keep state

这两条就是让从连接局域网的网卡上进入的数据平均分摊到两条出口上。所以我做了如下修改：

pass in quick on $int_if route-to ($ext_if2 $ext_gw2) proto tcp from $lan_net to 《表名》 flags S/SA modulate state

pass in quick on $int_if route-to ($ext_if1 $ext_gw1) proto tcp from $lan_net to any flags S/SA modulate state

现在似乎是成功了。
下面是我最终的修改结果，让 tcp 和 icmp 分线走，udp 到 dns 的走默认线路。其实我想可以在局域网网关机器上添加默认路由并做 DNS 转发，这样就可以都分开了。

# cat pf.2
lan_net = "192.168.0.0/24"
int_if = "rl0"
ext_if1 = "fxp0"
ext_if2 = "fxp1"
ext_gw1 = "172.16.11.1"
ext_gw2 = "172.16.12.1"
table persist file "/etc/telecom_ip"

# nat outgoing connections on each internet interface
### nat on $ext_if1 from $lan_net to any -> ($ext_if1)
### nat on $ext_if2 from $lan_net to any -> ($ext_if2)

nat on $ext_if2 from $lan_net to -> ($ext_if2)
nat on $ext_if1 from $lan_net to any -> ($ext_if1)

# default deny
block in from any to any
block out from any to any

# pass all outgoing packets on internal interface
pass out on $int_if from any to $lan_net

# pass in quick any packets destined for the gateway itself
pass in quick on $int_if from $lan_net to $int_if

# load balance outgoing tcp traffic from internal network.
#### pass in on $int_if route-to \
{ ($ext_if1 $ext_gw1), ($ext_if2 $ext_gw2) } round-robin \
proto tcp from $lan_net to any flags S/SA modulate state
############

pass in quick on $int_if route-to ($ext_if2 $ext_gw2) \
proto tcp from $lan_net to flags S/SA modulate state

pass in quick on $int_if route-to ($ext_if1 $ext_gw1) \
proto tcp from $lan_net to any flags S/SA modulate state

# load balance outgoing udp and icmp traffic from internal network
#### pass in on $int_if route-to \
{ ($ext_if1 $ext_gw1), ($ext_if2 $ext_gw2) } round-robin \
proto { udp, icmp } from $lan_net to any keep state
############

pass in on $int_if route-to ($ext_if1 $ext_gw1) \
proto udp from $lan_net to any port 53 keep state

pass in quick on $int_if route-to ($ext_if2 $ext_gw2) \
proto icmp from $lan_net to keep state

pass in quick on $int_if route-to ($ext_if1 $ext_gw1) \
proto icmp from $lan_net to any keep state


# general "pass out" rules for external interfaces
pass out on $ext_if1 proto tcp from any to any flags S/SA modulate state
pass out on $ext_if1 proto { udp, icmp } from any to any keep state
pass out on $ext_if2 proto tcp from any to any flags S/SA modulate state
pass out on $ext_if2 proto { udp, icmp } from any to any keep state

# route packets from any IPs on $ext_if1 to $ext_gw1 and the same for
# $ext_if2 and $ext_gw2
pass out on $ext_if1 route-to ($ext_if2 $ext_gw2) from $ext_if2 to 《表名》
pass out on $ext_if2 route-to ($ext_if1 $ext_gw1) from $ext_if1 to any

昨天晚上还看了一篇在 Route OS 上做双线的文章，它的优势是可以检测线路状态，当一条线路断掉了，就把双线并成一条线路来走。这个我想也可以写成一个脚本来实现。但是今天没时间了。

发表在这里的文字中的尖括号会被过滤，所以请以理解的方式阅读。Sorry。