<?xml version="1.0" encoding="UTF-8"?>
<!--
<!DOCTYPE article PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
"http://docbook.org/xml/4.2/docbookx.dtd">
-->

<!DOCTYPE article PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
"/usr/share/sgml/docbook/xml-dtd-4.2-1.0-22.1/docbookx.dtd">

<article id="index">
  <articleinfo>
    <title>Linux Route-based QoS mini HOWTO</title>

    <author>
      <firstname>John</firstname>
      <surname>Reuning</surname>
      <affiliation>
        <address><email>john-at-ibiblio-dot-org</email></address>
      </affiliation>
    </author>

    <pubdate>2004-03-31</pubdate>

    <revhistory>
      <revision>
        <revnumber>1.0</revnumber>

        <date>2004-03-31</date>

        <authorinitials>jrr</authorinitials>

        <revremark>first official release</revremark>
      </revision>
    </revhistory>

    <abstract>
      <para>This document describes how to combine Zebra (now Quagga) with
      iproute2 to achieve route-based QoS. It's somewhat of a hack, but it
      works well for providing different levels of service for a limited
      number of route classes. The goal for the given examples was to allow
      higher traffic levels for Internet2 destinations than for commodity
      Internet destinations.</para>
    </abstract>
  </articleinfo>

  <section id="intro">
    <title>Introduction</title>

    <para>This document describes how to combine <ulink
    url="http://www.zebra.org/">Zebra</ulink> (now <ulink
    url="http://www.quagga.net/">Quagga</ulink>) with iproute2 to achieve
    route-based QoS. It's somewhat of a hack, but it works well for providing
    different levels of service for a limited number of route classes. The
    goal for the given examples was to allow higher traffic levels for
    Internet2 destinations than for commodity Internet destinations. Much of
    the information contained in this document can be found in the <ulink
    url="http://lartc.org/">Linux Advanced Routing &amp; Traffic Contro
    HOWTO</ulink></para>

    <section id="copyright">
      <title>Copyright and License</title>

      <para>This document, <emphasis>Linux Route-based QoS HOWTO</emphasis>,
      is copyrighted (c) 2004 by <emphasis>John Reuning</emphasis>. Permission
      is granted to copy, distribute and/or modify this document under the
      terms of the GNU Free Documentation License, Version 1.1 or any later
      version published by the Free Software Foundation; with no Invariant
      Sections, with no Front-Cover Texts, and with no Back-Cover Texts. A
      copy of the license is available at <ulink
      url="http://www.gnu.org/copyleft/fdl.html">
      http://www.gnu.org/copyleft/fdl.html</ulink>.</para>

      <para>Linux is a registered trademark of Linus Torvalds.</para>
    </section>

    <section id="disclaimer">
      <title>Disclaimer</title>

      <para>No liability for the contents of this document can be accepted.
      Use the concepts, examples and information at your own risk. There may
      be errors and inaccuracies, that could be damaging to your system.
      Proceed with caution, and although this is highly unlikely, the
      author(s) do not take any responsibility.</para>

      <para>All copyrights are held by their by their respective owners,
      unless specifically noted otherwise. Use of a term in this document
      should not be regarded as affecting the validity of any trademark or
      service mark. Naming of particular products or brands should not be seen
      as endorsements.</para>
    </section>

    <section id="credits">
      <title>Credits / Contributors</title>
      <para>This document was converted to DocBook format with the help of 
      the <ulink url="http://www.tldp.org">Linux Documentation Project</ulink>'s
      <ulink url="http://tldp.org/authors/template/Sample-HOWTO.xml">
      Sample HOWTO template</ulink> and the 
      <ulink url="http://www.tldp.org/LDP/LDP-Author-Guide/html/index.html">
      LDP Author Guide</ulink>.
      </para>
    </section>

    <section id="feedback">
      <title>Feedback</title>

      <para>Feedback is most certainly welcome for this document. Send your
      additions, comments and criticisms to the following email address :
      <email>john-at-ibiblio-dot-org</email>.</para>
    </section>
  </section>

  <section id="what-is-qos">
    <title>What is route-based QoS?</title>

     <para>
     QoS is short for Quality of Service. For data networking, it refers
     to a set of properties that denote expectations for transport. It
     essentially covers network resource management. QoS can define levels for
     throughput, latency, jitter, etc. For example, multimedia streaming &amp;
     voice applications are more sensitive to latency than data transfer
     applications. QoS is sometimes used to ensure that the right kind of
     network resources are present to provide good service for these
     applications.
     </para>

     <para>
     In this case, however, QoS refers to controlling bandwidth usage 
     levels. The "route-based" of route-based QoS involves setting QoS classes 
     based on the destination of outbound network traffic.
     </para>
  </section>

  <section id="route-config">
    <title>Route Configuration</title>

    <section id="getting-the-routes">
      <title>Getting the routes</title>
      <para>
        The first step in the setup process is to get the routing tables. The 
        example requires the Internet2 BGP routing table via a route reflection 
        setup from a backbone Internet service provider. Adaptation of this 
        concept should be possible provided that multiple destination routes 
        exist and can be propogated to the Linux routing system.
      </para>

      <para>
        Working with Zebra or Quagga can be daunting if you're not familiar 
        with router configuration. The sample configuration files included 
        with the source code distribution are very helpful, and the interactive 
        interface is similar to that of Cisco routers. Below are sample Zebra 
        config files. If the tty interfaces to zebrad and bgpd are enabled, 
        be careful that they're protected. Use a hardware firewall or local 
        iptables.
      </para>
    </section>

    <section id="zebra-config">
      <title>Sample Quagga/Zebra configuration</title>
      <example><title>zebra.conf</title>
      <programlisting>
<![CDATA[
! zebra.conf
!
hostname director
password xxxxxx
enable password xxxxxx
log syslog
!log stdout
!log file zebra.log
!
smux peer .1.3.6.1.4.1.3317.1.2.1 zebra
!
]]>
      </programlisting>
      </example>

      <example><title>bgpd.conf</title>
      <programlisting>
<![CDATA[
! bgpd.conf
!
hostname bgpd
password xxxxxx
enable password xxxxxx
log syslog
!log stdout
!log file bgpd.log
smux peer .1.3.6.1.4.1.3317.1.2.2 zebra_bgpd
!
!bgp mulitple-instance
!
router bgp 2
  bgp router-id 192.168.1.254
  neighbor 10.0.1.1 remote-as 2
  neighbor 192.168.2.1 remote-as 2
  neighbor 192.168.2.1 route-reflector-client
  neighbor 192.168.2.2 remote-as 2
  neighbor 192.168.2.2 route-reflector-client
!
line vty
!
]]>
      </programlisting>
      </example>
    </section>

    <section id="check-the-routing">
      <title>Check the routing table</title>

      <para>
        Note that this configuration passes the BGP routes to other hosts. 
        Once you have the routing table, it's easy to give it to other 
        servers on your network.
      </para>

      <para>
        When Zebra is running, the system routing table should contain 
        lots of entries like this.
      </para>

      <programlisting>
<![CDATA[
10.0.0.0     default-router  255.255.255.0   UG    4475   0        0 eth0
]]>
      </programlisting>

      <para>
        This indicates that the dynamic routes are visible, so we move 
        on to the next step. Either of the following commands should 
        display the routing table:
      </para>

      <programlisting>
<prompt>$ </prompt><userinput><command>route</command></userinput>
<prompt>$ </prompt><userinput><command>ip</command>route show table main </userinput>
      </programlisting>
    </section>
  </section>

  <section id="route-transfer">
    <title>Transferring dynamic routes into a static routing table</title>

    <section id="copying-routes">
      <title>Copying routes into a static table</title>

      <para>
        For the route-based QoS to work, the dynamic routes must be connected to 
        the iproute2 filters. There exists a route filter that classifies packets 
        based on the destination's route realm value. The route-based packet 
        classification would be very easy if Zebra allowed setting a route id, 
        but this feature doesn't exist. The solution is to take the routing table, 
        copy the entries to a separate table, set a realm value for each in the process, 
        then add a lookup rule for the new routing table. The route filter can then 
        see the id of the destination route and classify outbound packets accordingly.
      </para>
    </section>

    <section id="route-refresh-script">
      <title>Example route refresh script</title>

      <example><title>refresh-qos-routes.pl</title>
      <programlisting>
<![CDATA[
#!/usr/bin/perl -w

use strict;
use Getopt::Long;
Getopt::Long::Configure('bundling', 'no_ignore_case');

my(
$opt_v,$verbose,
$opt_q,$quiet
);

#------------------------------------------------------------#
# read conf file arg
GetOptions (
	"v|verbose"		=> \$opt_v,
	"q|quiet"		=> \$opt_q
);

($opt_v) ? ($verbose = 1) : ($verbose = 0);
($opt_q) ? ($quiet = 1) : ($quiet = 0);

my $ipcmd = "/sbin/ip";
my $table = 99;

#------------------------------------------------------------#

if ($> != 0) {
        print "Only root can modify routes.  Exiting.\n";
        exit 1;
}

#------------------------------------------------------------#

# remove lookup rule
if (!$quiet) {
	print "Removing lookup rule for table $table\n";
}
system("$ipcmd rule del table $table");

# flush the existing routing table
if (!$quiet) {
	print "Flushing entries in table $table\n";
}
system("$ipcmd route flush table $table");

# grab the routes out of the main table
if (!$quiet) {
	print "Reading entries for main routing table\n";
}
my @list = `$ipcmd route show table main | grep 'zebra'`;

# add the I2 routes 
if (!$quiet) {
	print "Adding I2 entries to table $table\n";
}
my $item;
for $item (@list) {
	chomp($item);
	if ($verbose) {
		print "Adding: $item\n";
	}
	system("$ipcmd route add table $table $item realm 5");
}

# add static routes
if (!$quiet) {
	print "Adding static routes\n";
}
system("$ipcmd route add table $table 192.168.2.0/8 via 192.168.254.254 dev eth0 proto zebra realm 2");
system("$ipcmd route add table $table 192.168.3.0/8 via 192.168.254.254 dev eth0 proto zebra realm 2");
system("$ipcmd route add table $table 192.168.4.0/8 via 192.168.254.254 dev eth0 proto zebra realm 2");

# add lookup rule
system("$ipcmd rule add table $table");
]]>
      </programlisting>
      </example>
    </section>

    <section id="route-realm-value">
      <title>Route realm value</title>
      <para>
        This script could be run every few hours via cron. In the example 
        scenario, the Internet2 BGP routes do not change very frequently, 
        so the lack of dynamic updates is not a serious problem. The key task 
        of the script is to take routes added to the main routing table by Zebra 
        and copy them into the routing table 99 with a route realm value of 5. 
        Local network destinations are served by manually adding routes to cover 
        local subnets. These receive a realm value of 2. The realm values are 
        arbitrary, but it's important that the numbers match here and in the 
        packet filter configuration described below.
      </para>

      <para>
        [<ulink url="http://linux-ip.net/gl/ip-cref/node153.html">
        route realm explanation</ulink>]
      </para>
    </section>

  </section>


  <section id="traffic-filters">
    <title>Configuring the traffic filters and queues</title>

    <para>
      After the secondary route lookup table is in place, the next step 
      is to configure the packet queues and filters. The iproute2 Linux 
      kernel modules and tools are described in detail in the 
      <ulink url="http://lartc.org/">
      Linux Advanced Routing &amp; Traffic Contro HOWTO</ulink>.
    </para>

    <section id="htb-sfq-queues">
      <title>HTB and SFQ packet queues</title>
      <para>
        The HTB (Hierarchical Token Bucket) queue is used because it scales 
        well and provides a nice default queue feature. Designating a default 
        queue dumps packets into a queue if they aren't specifically filtered 
        and put into another queue. SFQ (Stochastic Fairness Queueing) is added 
        to each of the HTB queues. SFQ offers a reasonable level of equality 
        (among different flows) for limiting traffic while keeping the overhead 
        low. According to the documentation, it works well for queues that are 
        consistently full. The <command>route</command> packet classifier is 
        used to assign outbound 
        packets to queues based on the realm from the routing table.
      </para>
    </section>

    <section id="sample-traffic-script">
      <title>Sample traffic shaping script</title>
      <para>
        The following is a sample traffic shaping init script. These configurations 
        correspond to those used in the <command>refresh-qos-routes.pl</command> 
        script described above.
      </para>

      <example><title>traffic.sh</title>
      <programlisting>
<![CDATA[
#!/bin/sh
#
# traffic - script that configures network traffic shaping

extif=eth0

# line speed of the network interface
ifrate=1000mbit

# maximum traffic rate
maxrate=500mbit

# shaped limits
localrate=500mbit
i2rate=300mbit
i1rate=35mbit

TC=/sbin/tc

start() {
	# clear existing rules
	$TC qdisc del dev $extif root 2>/dev/null

	# root qdisc 1:0
	$TC qdisc add dev $extif root handle 1: \
		htb default 12

	# root class 1:1
	$TC class add dev $extif parent 1:0 classid 1:1 \
		htb rate $maxrate

	# class 1:10 -- local destinations
	$TC class add dev $extif parent 1:1 classid 1:10 \
		htb rate $localrate

	# class 1:11 -- I2 destinations
	$TC class add dev $extif parent 1:1 classid 1:11 \
		htb rate $i2rate

	# class 1:12 -- non-I2 destinations
	$TC class add dev $extif parent 1:1 classid 1:12 \
		htb rate $i1rate

	# qdisc defs for classes
	$TC qdisc add dev $extif parent 1:10 handle 10: \
		sfq quantum 1514b perturb 15
	$TC qdisc add dev $extif parent 1:11 handle 11: \
		sfq quantum 1514b perturb 15
	$TC qdisc add dev $extif parent 1:12 handle 12: \
		sfq quantum 1514b perturb 15

	# filter for 1:10 -- local destinations
	$TC filter add dev $extif parent 1:0 protocol ip pref 100 \
		route to 2 flowid 1:10

	# filter for 1:11 -- I2 routes
	$TC filter add dev $extif parent 1:0 protocol ip pref 100 \
		route to 5 flowid 1:11
}

stop() {
	# clear existing rules
	$TC qdisc del dev $extif root 2>/dev/null
}

status() {
	echo "qdisc:"
	$TC qdisc show dev $extif
	echo "filter:"
	$TC filter show dev $extif parent 1:
	echo "class:"
	$TC class show dev $extif
}

counts() {
	echo "qdisc:"
	$TC -s qdisc show dev $extif
	echo "class:"
	$TC -s class show dev $extif
}

case "$1" in
        start)
            start
            ;;
        
        stop)
            stop
            ;;
        
        status)
            status
            ;;
        counts)
            counts
            ;;
        restart)
            stop
            start
            ;;
        *)
            echo $"Usage: $0 {start|stop|restart|status|counts}"
            exit 1
esac

exit 0
]]>
      </programlisting>
      </example>
    </section>

    <section id="sample-script-explanation">
      <title>Sample traffic shaping script explained</title>
      <para>
        Four packet queues are created in the sample script -- one root queue 
        and three child queues. Each queue is assigned a maximum bandwidth. 
        When the queue rate limit is reached, outbound traffic is forcibly 
        slowed. Since TCP-based applications adjust, this is very effective 
        and doesn't sever the connections.
      </para>

      <para>
        The filtering, which places packets into the appropriate queue, is based 
        on the realm value assigned by the <command>refresh-qos-routes.pl</command> 
        script. The script assigned realm value 5 to the Internet2 routes, and realm 2 to 
        the local network routes. Thus, the route filter puts packets that match 
        a local network route into the high bandwidth queue. Internet2 destination 
        route packets go into the Internet2 queue. Any packet that doesn't match a 
        route is dumped into the default HTB queue, which has a lower traffic limit.
      </para>
    </section>
  </section>

  <section id="flow-graphing">
    <title>Graphing the flows</title>

    <para>
        At this point, the route-based QoS configuration is complete. However, 
        everyone likes to see colorful graphs of what the servers are doing. 
        Thus, here's how to use an MRTG or RRD tool to graph the outbound traffic 
        flows. The script below gathers queue counter data, converts it into bit 
        counts, and makes it available to snmpd.
    </para>

    <section id="data-gathering-script">
      <title>Sample data gathering script</title>

      <example><title>trafgraf.pl</title>
      <programlisting>
<![CDATA[
#!/usr/bin/perl -w

# trafgraf.pl
# a bridge between the iproute2 tool tc and snmp

# base source: http://www.docum.org/stef.coene/qos/gui/
#	but *heavily* modified

use strict;

sub nics ();
sub disp ($$);

#------------------------------------------------------------#
my $base = "/opt/trafgraf";
my $tccmd = "/sbin/tc";
my $ipcmd = "/sbin/ip";

my $debug = 0;

my ($key,%bytes,%tokens,$number);
my %parent;
my %nics = nics() ;

my $reqtype = "";
my $reqif = "";
my $reqqdisc = "";

#------------------------------------------------------------#
if ($#ARGV != 1) {
	# not called from snmp
} else {
	# set reqtype 'walk' or 'get'
	if ($ARGV[0] eq '-g') {
		$reqtype = "get";
	} elsif ($ARGV[0] eq '-n') {
		$reqtype = "walk";
	} else {
		$reqtype = "";
	}

	($reqif,$reqqdisc) = $ARGV[1] =~ /^\.1\.3\.6\.1\.4\.1\.2021\.255\.(\d+)\.*(\d*)/;
}



#------------------------------------------------------------#
# get all the info from all interfaces (qdisc + class)

my (@qdisc,@class,$line,$interface);
my ($byts,$pkts,$drops,$olimits);

foreach $interface (keys(%nics)) {
	$interface = $nics{$interface};
	@qdisc = `$tccmd -s -d qdisc show dev $interface`;
	@class = `$tccmd -s -d class show dev $interface`;
	$number = "";

	if ($debug) {
		print "start interface: $interface\n";
	}

	foreach $line (@qdisc) {
		chomp($line);
		if ($line =~ /^\s*$/) { next; }

		if ($debug) {
			print "qdisc line: $line\n";
		}

		# get qdisc id
		# qdisc sfq 8045: quantum 1514b limit 128p flows 128/1024
		if ($line =~ /^\s*qdisc/) {
			($number) = $line =~ /\s+([A-Za-z0-9]+:)\s+/o;
		}

		# Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
		if (($line =~ /^\s*Sent/) && $number) {
			($byts,$pkts,$drops,$olimits) = $line =~ /Sent\s+(\d+)\s+bytes\s+(\d+)\s+pkts\s+\(dropped\s+(\d+),\s+overlimits\s+(\d+)\)/;
			$bytes{"$interface $number"} = "$byts $pkts $drops $olimits";
			if ($debug) {
				print "$interface $number " . $bytes{"$interface $number"} . "\n";
			}
			$number = "";
		}
	}

	# let's skip the class info
	next;

	foreach $line (@class) {
		chomp ($line);
		if ($line =~ /^\s*$/) { next; }

		if ($debug) {
			print "class line: $line\n";
		}

		$line =~ /\s+([A-Za-z0-9]+:[A-Za-z0-9]+)\s+root/o 
			and $number = $1
			and $parent{"$interface $number"} = (split(":",$number))[0] . ":" ;

		$line =~ /\s+([A-Za-z0-9]+:[A-Za-z0-9]+)\s+parent\s+([A-Za-z0-9]+:[A-Za-z0-9]+)\s+leaf\s+([A-Za-z0-9]+:)\s+/o 
			and $number = $1
			and $parent{"$interface $number"} = $2
			and $parent{"$interface $3"} = $number
			and next ;

		$line =~ /\s+([A-Za-z0-9]+:[A-Za-z0-9]+)\s+parent\s+([A-Za-z0-9]+:[A-Za-z0-9]+)\s+/o 
			and $number = $1
			and $parent{"$interface $number"} = $2
			and next ;

		$line =~ /Sent\s+(\d+)\s+bytes\s+(\d+)\s+pkts\s+\(dropped\s+(\d+),\s+overlimits\s+(\d+)\)/
			and $bytes{"$interface $number"} = "$1 $2 $3 $4"
			and next ;

		$line =~ /lended:\s+(\d+)\s+borrowed:\s+(\d+)\s+giants:\s+(\d+)/
			and $lended{"$interface $number"} = "$1 $2 $3"
			and next ;

		$line =~ /tokens:\s+(\d+)\s+ctokens:\s+(\d+)/
			and $tokens{"$interface $number"} = "$1 $2" 
			and next ;
	}
}

#------------------------------------------------------------#
# figure out what to send back

# probably run manually, don't try and format results
if ($#ARGV < 1) {
	if ($debug) {
		print "Exiting due to manual run\n";
	}
	exit;
}

if (($reqtype eq "get") && ($ARGV[1] eq ".1.3.6.1.4.1.2021.255")) {
	if ($debug) {
		print "Exiting due reqtype == get\n";
	}
	exit;
}

# .1.3.6.1.4.1.2021.255.<interface>.<qdisc>

if ($debug) {
	print "reqif: $reqif\n";
	print "reqqdisc: $reqqdisc\n";
}

if (!defined($reqif)) {
	if ($debug) {
		print "Oops, no request interface\n";
	}
	exit;
}

if ($reqqdisc) {
	disp($reqif,$reqqdisc);
} else {
	disp($reqif,0);
}

exit;


#------------------------------------------------------------#
# print output
sub disp($$) {
	my $if = shift();
	my $id = shift();
	my $key = "eth" . $if . " $id:";
	my ($val,@keys,@tokens,$count);

	# if the id is 0, check for a 0: class
	if (($id == 0) && (!exists($bytes{$key}))) {
		@keys = sort(keys(%bytes));
		$key = $keys[0];
		($id) = $key =~ /^eth\d+ (\d+):$/;
	}

	$val = $bytes{$key};

	if ($debug) {
		print "bytes{$key} => $bytes{$key}\n";
	}

	if (defined($val)) {
		@tokens = split(/\s+/,$val);
		$count = $tokens[0];
	} else {
		$count = 0;
	}

	# adjust for 32 bit counter
	if ($count > 4294967295) {
		$count = $count % 4294967295;
	}

	print ".1.3.6.1.4.1.2021.255.$if.$id\n";
	print "counter\n";
	print "$count\n";
}

#------------------------------------------------------------#
# Find all configured nics
sub nics() {
	my @niclist = `$ipcmd link show`;
	my (%nic,$line);
	foreach $line (@niclist) {
		$line =~ /^([0-9]+):[ ]+([a-z0-9]+)/ or next;
		$nic{$1} = $2;
	}
	return(%nic);
}
]]>
      </programlisting>
      </example>
    </section>

    <section id="snmpd-conf-file">
      <title><command>snmpd</command> configuration</title>

      <para>
        The following lines should be appended to <command>snmpd.conf</command>
      </para>

      <programlisting>
<![CDATA[
# zebra & bgpd smux configuration

smuxpeer 1.3.6.1.4.1.3317.1.2.1 zebra
smuxpeer 1.3.6.1.4.1.3317.1.2.2 zebra_bgpd

# qos traffic stats
pass .1.3.6.1.4.1.2021.255 /opt/trafgraf/bin/snmp.pl
]]>
      </programlisting>
    </section>

    <section id="sample-cricket-graph">
      <title>Sample traffic flow graph</title>
      <mediaobject>
        <imageobject>
          <imagedata fileref="route-qos-graph.png" format="PNG"/>
        </imageobject>
        <textobject>
          <phrase>Sample Cricket graph</phrase>
        </textobject>
        <caption>
          <para>Sample graph of traffic flows generated using Cricket
          </para>
        </caption>
      </mediaobject>

      <para>
        The graph shows three categories: green, purple, and dark blue. From 
        the example scripts, the green line is local network traffic, the 
        purple is Internet2, and the blue is any other destination (regular 
        commodity Internet traffic). The orange line represents combined 
        traffic from the three categories. The green and purple categories are 
        allocated higher levels of bandwidth than the blue flow.
      </para>
    </section>
  </section>

  <section id="performance">
    <title>Performance</title>

    <para>
      So, you ask, "How well does this perform?" The graph in the 
      previous section shows traffic spikes of 70 Mbps. However, 
      production systems at ibiblio.org have reached sustained traffic 
      in excess of 150 Mbps with 8000 entries in the routing table 
      (on a dual P4 with 2.5 GB ram).
    </para>
  </section>

</article>


