Under FreeBSD, just like normal delay nodes, end node (sometimes called per-link) traffic shaping uses IPFW to direct traffic into the proper Dummynet pipe. On each node in a duplex link or LAN, a set of ipfw rules and Dummynet pipes is setup. As traffic enters or leaves your node, ipfw looks at the packet and stuffs it into the proper Dummynet pipe. At the proper time, Dummynet takes the packet and sends it on its way.
Under Linux, end node traffic shaping is performed by the packet scheduler modules, part of the kernel NET3 implementation. Each packet is added to the appropriate scheduler queue tree and shaped as specified in your NS file. Note that Linux traffic shaping currently only supports the drop-tail queueing discipline; gred and red are not available yet.
To specify end node shaping in
your NS file, simply setup a normal link or LAN, and then mark it as
wanting to use end node traffic shaping. For example:
Please be aware though, that the kernels are different than the standard
ones in a couple of ways:
set link0 [$ns duplex-link $nodeA $nodeD 50Mb 0ms DropTail]
set lan0 [$ns make-lan "nodeA nodeB nodeC" 1Mb 100ms]
tb-set-endnodeshaping $link0 1
tb-set-endnodeshaping $lan0 1
tb-use-endnodeshaping 1
If you would like to specify non-shaped links, but perhaps control the
shaping parameters later (increase delay, decrease bandwidth, etc.)
after the experiment is swapped in, you can put this in your NS file:
tb-force-endnodeshaping 1
set link0 [$ns duplex-link $nodeA $nodeB 50Mb 0ms DropTail]
set link1 [$ns duplex-link $nodeA $nodeC 50Mb 0ms DropTail]
tb-set-multiplexed $link0 1
tb-set-multiplexed $link1 1
Without multiplexed links, your experiment would not be mappable since
there are no nodes that can support the two duplex links that nodeA
requires; there is only one physical interface. Using multiplexed links
however, the testbed software will assign both links on NodeA to one
physical interface. That is because each duplex link is only 50Mbs,
while the physical link (fxp0) is 100Mbs. Of course, if your
application actually tried to use more than 50Mbs on each multiplexed link,
there would be a problem; a flow using more than its share on link0
would cause packets on link1 to be dropped when they otherwise would
not be. (At this time, you cannot specify that a LAN use multiplexed
links)
set maxnodes 12
set router [$ns node]
for {set i 1} {$i <= $maxnodes} {incr i} {
set node($i) [$ns node]
set link($i) [$ns duplex-link $node($i) $router 30Mb 10ms DropTail]
tb-set-multiplexed $link($i) 1
}
# Turn on routing.
$ns rtproto Static
Since each node on Emulab.Net has four 100Mbs interfaces, the above
mapping would not be possible without the use of multiplexed links.
However, since each link is defined to use 30Mbs, by using multiplexed
links, the 12 links can be shared over the four physical
interfaces, without oversubscribing the 400Mbs aggregate bandwidth
available to the node that is assigned to the router. Note:
while it may sound a little like channel bonding, it is not!
ipfw add pipe 10 ip from any to any out xmit fxp0
which says that any packet going out on fxp0 should be stuffed into
pipe 10. Consider the case of a ping packet that traverses a duplex
link from NodeA to NodeB. Once the proper interface is choosen (based
on routing or the fact that the destination is directly connected),
the packet is handed off to ipfw, which determines that the interface
(fxp0) matches the rule specified above. The packet is then stuffed
into the corresponding Dummynet pipe, to emerge sometime later (based
on the traffic shaping parameters) and be placed on the wire. The
packet then arrives at NodeB. A ping reply packet is created and
addressed to NodeA, placed into the proper Dummynet pipe, and arrives
at NodeA. As you can see, each packet traversed exactly one Dummynet
pipe (or put another way, the entire ping/reply sequence traversed two
pipes).
ipfw add pipe 15 ip from any to any in recv fxp0
which says that any packet received on fxp0 should be stuffed into
pipe 15. The packet is later handed up to the application, or
forwarded on to the next hop, if appropriate.
ipfw add ... out xmit fxp0
ipfw add ... in recv fxp0
do not work because there are now multiple flows multiplexed onto the
interface (multiple IPs) and so there is no way to distinguish which
flow. Consider a duplex link in which we use the first rule above.
If the packet is not addressed to a direct neighbor, the routing
code lookup will return a nexthop, which does indicate the
flow, but because the rule is based simply on the interface (fxp0), all
flows match! Unfortunately, ipfw does not provide an interface for
matching on the nexthop address, but seeing as we are kernel hackers,
this is easy to deal with by adding new syntax to ipfw to allow
matching on nexthop:
ipfw add ... out xmit fxp0 nexthop 192.168.2.3:255.255.255.0
Now, no matter how the user alters the routing table, packets will be
stuffed into the proper pipe since the nexthop indicates which
directly connected virtual link the packet was sent over. The use of a
mask allows for matching when directly connected to a LAN (a simplification).
# implicitly sets up class 1:1
tc qdisc add dev eth0 root handle 1 plr 0.05
# attach to class 1:1 and tell the module the default place to send
# traffic is to class 2:1 (could attach filters to discriminate)
tc qdisc add dev eth0 parent 1:1 handle 2 htb default 1
# class 2:1 does the actual limiting
tc class add dev eth0 parent 2 classid 2:1 htb rate 50Mbit ceil 50Mbit
# attach to class 2:1, also implicitly creates class 3:1, and attaches
# a FIFO queue to it.
tc qdisc add dev eth0 parent 2:1 handle 3 delay usecs 20000
The incoming side setup commands will look the same, but with eth0
replaced by imq0. Also, we have to tell the kernel to send packets
coming into eth0 to imq0 (where they will be shaped):
iptables -t mangle -A PREROUTING -i eth0 -j IMQ --todev 0
A flood ping sequence utilizing eth0 (echo->echo-reply) would
experience a round trip delay of 40 ms, be restricted to 50Mbit, and
have a 10% chance of losing packets. The doubling of numbers is due
to shaping as packets go out, and come back in the interface.