Category Archives: Networking Tools

Level3’s Filtergen – Tips and Tricks

BGP customers of Level3 (AS3356) would do well to learn the intricacies of their filtergen system. Level3 uses filtergen to build route filters toward customer BGP sessions. You can accidentally affect what traffic gets exchanged over your AS3356 transit if you don’t manager your IRR records carefully.

Many other large networks operate their own customer-centric IRR’s too. However, I always recommend going with a 3rd party IRR. While Level3 customers are free to register their objects in Level3’s IRR, they may also use the IRR of their choice. RADB, RIPE, and ARIN are popular choices (The free ALTDB isn’t very well maintained anymore).

By default, Level 3 builds filters based on entries in their own LEVEL3 IRR. However Continue reading

Thruk with FreeBSD Nagios — MK Livestatus Over The Network Without inetd

At $dayjob, the Nagios server runs on FreeBSD. Rather than porting Thruk to FreeBSD, we run it on a separate Linux VM (and just use the Debian package). However, that poses some problems with backend availability.

The MK Livestatus documentation suggests using xinetd and gives an example. However, the FreeBSD server in question runs stock inetd, and I didn’t want to change it. Further, the stock inetd causes a new process to be spawned with each query. This results in occasional backend timeout issues on even a lightly-loaded Thruk installation.

So, instead of using [x]inetd, I went with socat. The config was pretty simple after installing from ports:

/etc/rc.conf:
socat_enable="YES"
socat_flags="TCP4-LISTEN:6557,fork,reuseaddr UNIX-CONNECT:/var/spool/nagios/rw/live"

The new socat method is more efficient and reliable that using inetd to spawn a new process for every socket connection.

Since Thruk is happy connecting with a network socket, I didn’t bother recreating the FreeBSD Nagios’s server’s domain socket over to the Linux Thruk server. However, doing so would simply have been a matter of using another socat configuration on the side of the Thruk server. i.e.

socat UNIX-LISTEN:/var/lib/thruk/live,fork,user=www-data,group=www-data TCP:[$NAGIOS_SERVER_IP]:6557

If I wanted too, I could even use socat’s SSL encryption over the wire to protect the contents of the livestatus shell.

Pro-Tip: Port Mirroring In Linux

Sometimes a span/mirror port isn’t available on a switch when you need one. However, if you just need to monitor traffic from/to a Linux server, you can have the Linux server mirror it’s own traffic toward a collector. You just need a spare NIC on the Linux system.

For example, if you want to mirror all traffic going in/out of eth0, and send it out eth1 toward a collector, you can do that with the ‘tc’ subsystem. Some examples of this exist online. However, what they don’t tell you is that if eth1 goes down (unplugged, etc), that results in eth0 blocking all traffic — not good! I came up with a workaround for that problem using bridge and dummy interfaces. It’s kludgy, but it works!

See below for all the required config files / scripts. (don’t forget to chmod the scripts so root can run them!)

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet static
        address 192.168.1.123
        netmask 255.255.255.20
        gateway 192.168.1.254
        dns-nameservers 8.8.8.8

# physical mirror port toward collector
auto eth1
iface eth1 inet static
        address 127.1.1.1
        netmask 255.255.255.255
        up ip addr del 127.1.1.1/32 dev eth1;:

# always-up mirror port toward dev null
auto dummy0
iface dummy0 inet static
        address 127.4.4.4
        netmask 255.255.255.255
        pre-up modprobe dummy;:
        up ip addr del 127.4.4.4/32 dev dummy0;:

# bridge of eth1 and dummy0
# this is so that it always stays up, even if eth1 is down
auto br0
iface br0 inet static
        bridge_ports eth1 dummy0
        address 127.5.5.5
        netmask 255.255.255.255
        up ip addr del 127.5.5.5/32 dev br0;:
        post-up /etc/network/mirror-up.sh;:
        pre-down /etc/network/mirror-down.sh;:
#!/bin/sh

# ALK 2014-10-23
# Send all 'source_if' traffic (ingress & egress) to collector box on 'dest_if' 

# Normally called by boot process or ifup. i.e.
# --/etc/network/interfaces--
#   iface eth0 inet static
#     address X.X.X.X
#     netmask Y.Y.Y.Y
#     post-up /etc/network/mirror-up.sh;:

source_if=eth0
dest_if=br0

# enable the destination port
ifconfig $dest_if up;:

# mirror ingress traffic
tc qdisc add dev $source_if ingress;:
tc filter add dev $source_if parent ffff: \
	protocol all \
	u32 match u8 0 0 \
	action mirred egress mirror dev $dest_if;:

# mirror egress traffic
tc qdisc add dev $source_if handle 1: root prio;:
tc filter add dev $source_if parent 1: \
	protocol all \
	u32 match u8 0 0 \
	action mirred egress mirror dev $dest_if;:
#!/bin/sh

# ALK 2014-10-23
# De-provision mirroring config (See mirror-up.sh for provisioning) 

# Normally called by boot process or ifdown. i.e.
# --/etc/network/interfaces--
#   iface eth0 inet static
#     address X.X.X.X
#     netmask Y.Y.Y.Y
#     pre-down /etc/network/mirror-down.sh;:

source_if=eth0

# de-provision ingress mirroring
tc qdisc del dev $source_if ingress;:

# de-provisoin egress mirroring
tc qdisc del dev $source_if root;:

ISC DHCPD – Putting Option-82 Vendor Codes To Use To Reformat Cirtuit-ID and Remote-ID Values

The broadband access network I work with relies on DHCP Option-82 to manage DHCP leases, among other things.

The primary access equipment vendor (Cambium — a spinoff of Motorola) is unfriendly, in that the Option-82 circuit-id (i.e. Access Point) and remote-id (i.e. Subscriber Module) fields are binary representations of the respective network element’s MAC addresses. Even `tcpdump’ has a hard time reading it:

Remote-ID SubOption 2, length 6: ^J^@>8ZM-`

That poses certain problems. For one, creating a static IP assignment (based on the customer SM’s MAC address, which is Option 82 remote-id) gets ugly in dhcpd.conf file. For example:

match if binary-to-ascii(16, 8, ":", suffix( option agent.remote-id, 6)) = "a:0:3e:38:5a:e0"

… is a lot harder to read than

match if option agent.remote-id = "a:0:3e:38:5a:e0"

But since agent.remote-id is a binary-encoded MAC address, rather than human-readable string, the binary-to-ascii conversion is necessary. And, since the binary-to-ascii function strips leading zeros, we have to revisit a bunch of existing code that tracks CPE MAC addresses. What I wanted was a way to convert the remote-id (SM’s MAC address) and circuit-id (AP’s MAC address) to a human-readable MAC addresses, to make dhcpd.conf file easier to read and script.

UPDATE: It turns out I could indeed match the agent.remote-id using a standard represtation of a MAC address for the remote-id. I just needed to leave out the quotes! For example:

match if option agent.remote-id = 0a:00:3e:38:5a:e0

Had I realized that, I never would have bothered with the decoding the Option 82 vendor string and all the other work I describe below. Oh well… at least I learned something.

Because the DHCP server may be used for other vendors (i.e. GPON, etc), I wanted to only do these manipulations of the Option-82 data fields that are set by the Motorola/Cambium radios. Fortunately, Cambium AP’s also set Option 82 Subclass 9, which a vendor specific entry.

While `tcpdump’ doesn’t understand 82.9:

Unknown SubOption 9, length 11:
0x0000: 0000 00a1 0613 0401 020b b8

… Wireshark does!

Option 82 Suboption: (9) Vendor-Specific Information
Length: 11
Enterprise: Motorola (161)
Data Length: 6
Value: 130401020bb8

Hex: 090b000000a106130401020bb8

Reading this:

82.9.1 is the 4-byte Vendor ID (i.e. IANA Enterprise assignments, like what is used in SNMP MIB’s)

82.9.2 is an unspecified field that is 6-bytes long in this case (I can’t figure out what, if anything, it translates too)

Unfortunately, as of dhcpd 4.3.1, there is no “option agent.vendor-id” field available — out of the 12 DHCP Option-82 subcodes currently defined, ISC’s man page for `dhcp-options` only lists four possibilities for agent matching:

  • 82.1: option agent.circuit-id
  • 82.2: option agent.remote-id
  • 82.4: option agent.DOCSIS-device-class
  • 82.5: option agent.link-selection

Fortunately, I found that the vendor (subcode 9) options are available — it’s just not documented. But if you know where to look, you can find that the undocumented ‘option agent.unknown-9’ field contains the raw bits of the vendor extension field. This seems easier than this workaround somebody else discovered — plus it doesn’t destory the ‘option agent.*’ information that’s used for the ‘stash-agent-option’ config flag.

Using ‘option agent.unknown-9’, I was able to grab the vendor ID (what Wireshark decoded as 161 — Motorola) in dhcpd.conf:

set myvendor = binary-to-ascii(10, 16, ",", substring(option agent.unknown-9,2,2));

Putting it all together:

# when the DHCP server gets an initial request, it contains Option 82 information
# that is added by a network element (i.e. from the AP, OLT, CMTS, DSLAM, switch, etc).
# however, subsequent renewals are unicasted from the subscriber to the DHCP server.
# those unicasted requests don't get Option 82 values added by the network element.
# because we will match on agent options, we need to stash this info so that renewals
# will work as expected. basically, this will create a database linking the hardware address
# to the Option 82 values that were found in the initial DHCP DISCOVER packet
stash-agent-option;

# DHCP Option 82 *may* contain a vendor subtype (code 9)
# ISC-DHCPD knows it's there, but doesn't know how to decode it
if exists agent.unknown-9
{

# VENDOR definition
set myvendor = binary-to-ascii(10, 16, ",", substring(option agent.unknown-9,2,2));

# For customers behind Motorola (i.e. Cambium SM), format ciruit-id and remote-id as MAC's
if myvendor = "161" # Motorola
{
# SM definition
# we want to restore the leading zero's that get stripped when converting binary to ascii
set mysm = concat (
suffix (concat ("0", binary-to-ascii (16, 8, "", substring(option agent.remote-id,0,1))),2), ":",
suffix (concat ("0", binary-to-ascii (16, 8, "", substring(option agent.remote-id,1,1))),2), ":",
suffix (concat ("0", binary-to-ascii (16, 8, "", substring(option agent.remote-id,2,1))),2), ":",
suffix (concat ("0", binary-to-ascii (16, 8, "", substring(option agent.remote-id,3,1))),2), ":",
suffix (concat ("0", binary-to-ascii (16, 8, "", substring(option agent.remote-id,4,1))),2), ":",
suffix (concat ("0", binary-to-ascii (16, 8, "", substring(option agent.remote-id,5,1))),2)
);

# AP definition
# we want to restore the leading zero's that get stripped when converting binary to ascii
set myap = concat (
suffix (concat ("0", binary-to-ascii (16, 8, "", substring(option agent.circuit-id,0,1))),2), ":",
suffix (concat ("0", binary-to-ascii (16, 8, "", substring(option agent.circuit-id,1,1))),2), ":",
suffix (concat ("0", binary-to-ascii (16, 8, "", substring(option agent.circuit-id,2,1))),2), ":",
suffix (concat ("0", binary-to-ascii (16, 8, "", substring(option agent.circuit-id,3,1))),2), ":",
suffix (concat ("0", binary-to-ascii (16, 8, "", substring(option agent.circuit-id,4,1))),2), ":",
suffix (concat ("0", binary-to-ascii (16, 8, "", substring(option agent.circuit-id,5,1))),2)
);

# add a line to syslog showing the IP -> SM -> AP mapping
log ( info, concat (
"WISP IPv4 ", binary-to-ascii (10, 8, ".", leased-address),
" SM " , mysm,
" AP ", myap,
" VENDOR ", myvendor));
}

}

# to simplify later 'deny' rules, track all static customers here
class "static-sm" {
# unfortunately, I can't just do a "match mysm" here, so I have to redo my string manipulation on the "option agent.remote-id"
match concat (
suffix (concat ("0", binary-to-ascii (16, 8, "", substring(option agent.remote-id,0,1))),2), ":",
suffix (concat ("0", binary-to-ascii (16, 8, "", substring(option agent.remote-id,1,1))),2), ":",
suffix (concat ("0", binary-to-ascii (16, 8, "", substring(option agent.remote-id,2,1))),2), ":",
suffix (concat ("0", binary-to-ascii (16, 8, "", substring(option agent.remote-id,3,1))),2), ":",
suffix (concat ("0", binary-to-ascii (16, 8, "", substring(option agent.remote-id,4,1))),2), ":",
suffix (concat ("0", binary-to-ascii (16, 8, "", substring(option agent.remote-id,5,1))),2)
);
}

# each static-IP subscriber gets a "static-sm" subclass entry like this
subclass "static-sm" "0a:00:3e:38:5a:e0";

# each static-IP subscriber gets a unique class with just their SM's MAC
class "alk-static" {
match if mysm = "0a:00:3e:38:5a:e0";
}

# BRAS/BNG Testing Subnets
shared-network brastest-default {
subnet 192.168.1.0 netmask 255.255.255.0 {
option routers 192.168.1.254;
}
pool {
# this is the static IP assignment for "alk-static"
# each static IP customer gets a pool definition like this one
allow members of "alk-static";
allow known-clients;
range 192.168.1.200;
}
pool {
# this is the default pool for dynamic customers
deny dynamic bootp clients; # DHCP is OK, BOOTP is not...
deny members of "static-sm"; # List of all static assignments (subclasses)
range 192.168.1.1 192.168.1.199;
}
}

ProTip: Forcing clean exit status on Linux (Bourne Shell) commands/scripts with “;:”

Often times in Linux (or BSD, Unix, etc) you want to trigger external commands, but don’t care if they succeed or fail. For example, in Debian, you can execute scripts and commands from /etc/network/interfaces using the [pre-]up and [post-]down directives. However, if something goes wrong and the command doesn’t exit cleanly, the interface action (ifup or ifdown) will fail. This can happen, for example, when adding a static route (ip route add ...) or alias address (ip addr add ...) that already exists ("RTNETLINK answers: File exists"). Another common scenario is using 'ethtool' to set interface parameters such as Tx/Rx queue sizes. If the ethtool command “fails” because the value is already set, that will cause the 'ifup' or 'ifdown' command to fail as well.

For example Continue reading

A utility to perform IPv4 & IPv6 prefix aggregation

I’m a fan of the aggregate program, which, in its own words, is used to “optimise a list of route prefixes to help make nice short filters”. However, it doesn’t work for IPv6! It was written by Joe Abley back in the 1990’s, and is useful in lots of scripts, but the lack of IPv6 support is starting to be an issue for me.

Using the IPy python module, I’ve created a similar tool that handles both IPv4 and IPv6 prefixes. Like the old tool, it can handle host addresses that don’t have a netmask specified, and can truncate the network portion to match the netmask if they are mismatched. I also added support for pre-1993 classful networking (since IOS and Quagga/Zebra still seem to leave the mask off such networks, two decades post-CIDR).

It doesn’t have all the features of the old aggregate command (i.e. –max-opt-lengh, etc), but it has all the features I used, and some that I had always wished for.

Continue reading

Using (Abusing?) AWK For Rudimentary DoS Detection

sFlow(R)InMon’s sflowtool comes with an sample script, ipTopTalkers that uses awk to provide a list of toptalkers based on sflow sampled packets. All I’ve done here is build on their ipTopTalkers sample code to find which addresses are receiving >10,000 pps or >80 Mb/s, and send that information to a log file and reporting-script (not shown).

Converting the binary sflow feed to text, then piping it through awk to do basic math and sorting functions, is not exactly the most elegent solution. But hey, it works!

#!/usr/bin/mawk-cur -f

# ALK 2013-01-10
#
# for performance reasons, mawk is preferred to gawk
# newer versions of mawk include things like strftime that tradionally had been
# gawk-only features. code and builds of mawk-cur (cutting edge) can be found at:
#    http://invisible-island.net/mawk/
# (the features of mawk-cur aren't available in official debian packages of mawk)
#
# based on ipTopTalkers from InMon:
#
# Copyright (c) 2001 InMon Corp. Licensed under the terms of the InMon sFlow licence:
# http://www.inmon.com/technology/sflowlicense.txt

# usage: sflowtool | DoSTargets

BEGIN{
  lastInt = 0;
  report = "tee -a /var/log/ddos-report.log | dos-report.sh";
  interval = 60; #1 minute window
  BPSthreshold = 83886080; # alert threshold in bits per second i.e. 80 Mb/s
  PPSthreshold = 10000; # alert threshold in packets per second i.e 10kpps
}
/unixSecondsUTC/{ 
  currentInt = $2 - ($2 % interval);
  if(currentInt != lastInt) {
    for(i = 1; i < = 1000; i++) { # consider up to 1000 simultaneous targets
      BPSmaxCount = 0;
      BPSmaxKey = "";
      for(BPSkey in BPScount) {
        if(BPScount[BPSkey] > BPSmaxCount) {
          BPSmaxCount = BPScount[BPSkey];
          BPSmaxKey = BPSkey;
        }
      }
      if(BPSmaxCount > (BPSthreshold * interval)) printf("%d %s %d %s", strftime("%s", lastInt), BPSmaxKey, sprintf("%d",(BPSmaxCount/1024/1024/interval)),"mbps\n") | report;
      delete BPScount[BPSmaxKey];

      PPSmaxCount = 0;
      PPSmaxKey = "";
      for(PPSkey in PPScount) {
        if(PPScount[PPSkey] > PPSmaxCount) {
          PPSmaxCount = PPScount[PPSkey];
          PPSmaxKey = PPSkey;
        }
      }
      if(PPSmaxCount > (PPSthreshold * interval)) printf("%d %s %d %s", strftime("%s", lastInt), PPSmaxKey, sprintf("%d",(PPSmaxCount/interval)),"pps\n") | report;
      delete PPScount[PPSmaxKey];
    }
    fflush(stdout); # write out stdout buffer
    close(report); # send the alert email
    lastInt = currentInt;
    delete BPScount;
    delete PPScount;
  }
}
/meanSkipCount/{ samplingInterval = $2; }
/sampledPacketSize/{ sampledPacketSize = $2; } 
/dstIP/{ BPScount[$2] = BPScount[$2] + ( samplingInterval * sampledPacketSize * 8); PPScount[$2] = PPScount[$2] + samplingInterval; }
END{}

U-Turn: Reverse Traceroute Using Scapy

Thug means never having to say you're sorry..

I’ve always wanted to write a reverse-traceroute system. However, writing my own Linux NetFilter module was daunting – I’m not a C programmer, let alone a kernel hacker. Then I came across Scapy.

While intercepting and mangling the packets in userspace does add some artificial latency, it is a good proof-of-concept. And of course, it supports both IPv4 and IPv6:

Continue reading