Author Archives: adamkuj

Level3’s Filtergen – Tips and Tricks

BGP customers of Level3 (AS3356) would do well to learn the intricacies of their filtergen system. Level3 uses filtergen to build route filters toward customer BGP sessions. You can accidentally affect what traffic gets exchanged over your AS3356 transit if you don’t manager your IRR records carefully.

Many other large networks operate their own customer-centric IRR’s too. However, I always recommend going with a 3rd party IRR. While Level3 customers are free to register their objects in Level3’s IRR, they may also use the IRR of their choice. RADB, RIPE, and ARIN are popular choices (The free ALTDB isn’t very well maintained anymore).

By default, Level 3 builds filters based on entries in their own LEVEL3 IRR. However Continue reading →

SSH Keyagent

While most desktop Linux OS’s have decent support for ssh-agent, it may be something you have to add to certain Linux or *BSD disto’s. I’ve been using the following to load my private key(s) the first time I login, and hold them in memory for subsequent sessions.

SSH_ENV="$HOME/.ssh/environment"

function start_agent {
echo "Initialising new SSH agent..."
(umask 066; /usr/bin/ssh-agent > "${SSH_ENV}")
. "${SSH_ENV}" > /dev/null
/usr/bin/ssh-add;
}

# Source SSH settings, if applicable

if [ -f "${SSH_ENV}" ]; then
. "${SSH_ENV}" > /dev/null
ps -ef | grep ${SSH_AGENT_PID} | grep ssh-agent$ > /dev/null || {
start_agent;
}
else
start_agent;
fi

if [ -z "$SSH_AUTH_SOCK" ]; then
eval ssh-agent -s ssh-add
fi

Thruk with FreeBSD Nagios — MK Livestatus Over The Network Without inetd

At $dayjob, the Nagios server runs on FreeBSD. Rather than porting Thruk to FreeBSD, we run it on a separate Linux VM (and just use the Debian package). However, that poses some problems with backend availability.

The MK Livestatus documentation suggests using xinetd and gives an example. However, the FreeBSD server in question runs stock inetd, and I didn’t want to change it. Further, the stock inetd causes a new process to be spawned with each query. This results in occasional backend timeout issues on even a lightly-loaded Thruk installation.

So, instead of using [x]inetd, I went with socat. The config was pretty simple after installing from ports:

/etc/rc.conf:
socat_enable="YES"
socat_flags="TCP4-LISTEN:6557,fork,reuseaddr UNIX-CONNECT:/var/spool/nagios/rw/live"

The new socat method is more efficient and reliable that using inetd to spawn a new process for every socket connection.

Since Thruk is happy connecting with a network socket, I didn’t bother recreating the FreeBSD Nagios’s server’s domain socket over to the Linux Thruk server. However, doing so would simply have been a matter of using another socat configuration on the side of the Thruk server. i.e.

socat UNIX-LISTEN:/var/lib/thruk/live,fork,user=www-data,group=www-data TCP:[$NAGIOS_SERVER_IP]:6557

If I wanted too, I could even use socat’s SSL encryption over the wire to protect the contents of the livestatus shell.

Pro-Tip: Port Mirroring In Linux

Sometimes a span/mirror port isn’t available on a switch when you need one. However, if you just need to monitor traffic from/to a Linux server, you can have the Linux server mirror it’s own traffic toward a collector. You just need a spare NIC on the Linux system.

For example, if you want to mirror all traffic going in/out of eth0, and send it out eth1 toward a collector, you can do that with the ‘tc’ subsystem. Some examples of this exist online. However, what they don’t tell you is that if eth1 goes down (unplugged, etc), that results in eth0 blocking all traffic — not good! I came up with a workaround for that problem using bridge and dummy interfaces. It’s kludgy, but it works!

See below for all the required config files / scripts. (don’t forget to chmod the scripts so root can run them!)

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet static
address 192.168.1.123
netmask 255.255.255.20
gateway 192.168.1.254
dns-nameservers 8.8.8.8

# physical mirror port toward collector
auto eth1
iface eth1 inet static
address 127.1.1.1
netmask 255.255.255.255
up ip addr del 127.1.1.1/32 dev eth1;:

# always-up mirror port toward dev null
auto dummy0
iface dummy0 inet static
address 127.4.4.4
netmask 255.255.255.255
pre-up modprobe dummy;:
up ip addr del 127.4.4.4/32 dev dummy0;:

# bridge of eth1 and dummy0
# this is so that it always stays up, even if eth1 is down
auto br0
iface br0 inet static
bridge_ports eth1 dummy0
address 127.5.5.5
netmask 255.255.255.255
up ip addr del 127.5.5.5/32 dev br0;:
post-up /etc/network/mirror-up.sh;:
pre-down /etc/network/mirror-down.sh;:

#!/bin/sh

# ALK 2014-10-23
# Send all 'source_if' traffic (ingress & egress) to collector box on 'dest_if'

# Normally called by boot process or ifup. i.e.
# --/etc/network/interfaces--
#   iface eth0 inet static
#     address X.X.X.X
#     netmask Y.Y.Y.Y
#     post-up /etc/network/mirror-up.sh;:

source_if=eth0
dest_if=br0

# enable the destination port
ifconfig $dest_if up;:

# mirror ingress traffic
tc qdisc add dev $source_if ingress;:
tc filter add dev $source_if parent ffff: \
protocol all \
u32 match u8 0 0 \
action mirred egress mirror dev $dest_if;:

# mirror egress traffic
tc qdisc add dev $source_if handle 1: root prio;:
tc filter add dev $source_if parent 1: \
protocol all \
u32 match u8 0 0 \
action mirred egress mirror dev $dest_if;:

#!/bin/sh

# ALK 2014-10-23
# De-provision mirroring config (See mirror-up.sh for provisioning)

# Normally called by boot process or ifdown. i.e.
# --/etc/network/interfaces--
#   iface eth0 inet static
#     address X.X.X.X
#     netmask Y.Y.Y.Y
#     pre-down /etc/network/mirror-down.sh;:

source_if=eth0

# de-provision ingress mirroring
tc qdisc del dev $source_if ingress;:

# de-provisoin egress mirroring
tc qdisc del dev $source_if root;:

ISC DHCPD – Putting Option-82 Vendor Codes To Use To Reformat Cirtuit-ID and Remote-ID Values

The broadband access network I work with relies on DHCP Option-82 to manage DHCP leases, among other things.

The primary access equipment vendor (Cambium — a spinoff of Motorola) is unfriendly, in that the Option-82 circuit-id (i.e. Access Point) and remote-id (i.e. Subscriber Module) fields are binary representations of the respective network element’s MAC addresses. Even `tcpdump’ has a hard time reading it:

Remote-ID SubOption 2, length 6: ^J^@>8ZM-`

That poses certain problems. For one, creating a static IP assignment (based on the customer SM’s MAC address, which is Option 82 remote-id) gets ugly in dhcpd.conf file. For example:

match if binary-to-ascii(16, 8, ":", suffix( option agent.remote-id, 6)) = "a:0:3e:38:5a:e0"

… is a lot harder to read than

match if option agent.remote-id = "a:0:3e:38:5a:e0"

But since agent.remote-id is a binary-encoded MAC address, rather than human-readable string, the binary-to-ascii conversion is necessary. And, since the binary-to-ascii function strips leading zeros, we have to revisit a bunch of existing code that tracks CPE MAC addresses. What I wanted was a way to convert the remote-id (SM’s MAC address) and circuit-id (AP’s MAC address) to a human-readable MAC addresses, to make dhcpd.conf file easier to read and script.

UPDATE: It turns out I could indeed match the agent.remote-id using a standard represtation of a MAC address for the remote-id. I just needed to leave out the quotes! For example:

match if option agent.remote-id = 0a:00:3e:38:5a:e0

Had I realized that, I never would have bothered with the decoding the Option 82 vendor string and all the other work I describe below. Oh well… at least I learned something.

Because the DHCP server may be used for other vendors (i.e. GPON, etc), I wanted to only do these manipulations of the Option-82 data fields that are set by the Motorola/Cambium radios. Fortunately, Cambium AP’s also set Option 82 Subclass 9, which a vendor specific entry.

While `tcpdump’ doesn’t understand 82.9:

Unknown SubOption 9, length 11:
0x0000: 0000 00a1 0613 0401 020b b8

… Wireshark does!

Option 82 Suboption: (9) Vendor-Specific Information
Length: 11
Enterprise: Motorola (161)
Data Length: 6
Value: 130401020bb8

Hex: 090b000000a106130401020bb8

Reading this:

82.9.1 is the 4-byte Vendor ID (i.e. IANA Enterprise assignments, like what is used in SNMP MIB’s)

82.9.2 is an unspecified field that is 6-bytes long in this case (I can’t figure out what, if anything, it translates too)

Unfortunately, as of dhcpd 4.3.1, there is no “option agent.vendor-id” field available — out of the 12 DHCP Option-82 subcodes currently defined, ISC’s man page for `dhcp-options` only lists four possibilities for agent matching:

  • 82.1: option agent.circuit-id
  • 82.2: option agent.remote-id
  • 82.4: option agent.DOCSIS-device-class
  • 82.5: option agent.link-selection

Fortunately, I found that the vendor (subcode 9) options are available — it’s just not documented. But if you know where to look, you can find that the undocumented ‘option agent.unknown-9’ field contains the raw bits of the vendor extension field. This seems easier than this workaround somebody else discovered — plus it doesn’t destory the ‘option agent.*’ information that’s used for the ‘stash-agent-option’ config flag.

Using ‘option agent.unknown-9’, I was able to grab the vendor ID (what Wireshark decoded as 161 — Motorola) in dhcpd.conf:

set myvendor = binary-to-ascii(10, 16, ",", substring(option agent.unknown-9,2,2));

Putting it all together:

# when the DHCP server gets an initial request, it contains Option 82 information
# that is added by a network element (i.e. from the AP, OLT, CMTS, DSLAM, switch, etc).
# however, subsequent renewals are unicasted from the subscriber to the DHCP server.
# those unicasted requests don't get Option 82 values added by the network element.
# because we will match on agent options, we need to stash this info so that renewals
# will work as expected. basically, this will create a database linking the hardware address
# to the Option 82 values that were found in the initial DHCP DISCOVER packet
stash-agent-option;

# DHCP Option 82 *may* contain a vendor subtype (code 9)
# ISC-DHCPD knows it's there, but doesn't know how to decode it
if exists agent.unknown-9
{

# VENDOR definition
set myvendor = binary-to-ascii(10, 16, ",", substring(option agent.unknown-9,2,2));

# For customers behind Motorola (i.e. Cambium SM), format ciruit-id and remote-id as MAC's
if myvendor = "161" # Motorola
{
# SM definition
# we want to restore the leading zero's that get stripped when converting binary to ascii
set mysm = concat (
suffix (concat ("0", binary-to-ascii (16, 8, "", substring(option agent.remote-id,0,1))),2), ":",
suffix (concat ("0", binary-to-ascii (16, 8, "", substring(option agent.remote-id,1,1))),2), ":",
suffix (concat ("0", binary-to-ascii (16, 8, "", substring(option agent.remote-id,2,1))),2), ":",
suffix (concat ("0", binary-to-ascii (16, 8, "", substring(option agent.remote-id,3,1))),2), ":",
suffix (concat ("0", binary-to-ascii (16, 8, "", substring(option agent.remote-id,4,1))),2), ":",
suffix (concat ("0", binary-to-ascii (16, 8, "", substring(option agent.remote-id,5,1))),2)
);

# AP definition
# we want to restore the leading zero's that get stripped when converting binary to ascii
set myap = concat (
suffix (concat ("0", binary-to-ascii (16, 8, "", substring(option agent.circuit-id,0,1))),2), ":",
suffix (concat ("0", binary-to-ascii (16, 8, "", substring(option agent.circuit-id,1,1))),2), ":",
suffix (concat ("0", binary-to-ascii (16, 8, "", substring(option agent.circuit-id,2,1))),2), ":",
suffix (concat ("0", binary-to-ascii (16, 8, "", substring(option agent.circuit-id,3,1))),2), ":",
suffix (concat ("0", binary-to-ascii (16, 8, "", substring(option agent.circuit-id,4,1))),2), ":",
suffix (concat ("0", binary-to-ascii (16, 8, "", substring(option agent.circuit-id,5,1))),2)
);

# add a line to syslog showing the IP -> SM -> AP mapping
log ( info, concat (
"WISP IPv4 ", binary-to-ascii (10, 8, ".", leased-address),
" SM " , mysm,
" AP ", myap,
" VENDOR ", myvendor));
}

}

# to simplify later 'deny' rules, track all static customers here
class "static-sm" {
# unfortunately, I can't just do a "match mysm" here, so I have to redo my string manipulation on the "option agent.remote-id"
match concat (
suffix (concat ("0", binary-to-ascii (16, 8, "", substring(option agent.remote-id,0,1))),2), ":",
suffix (concat ("0", binary-to-ascii (16, 8, "", substring(option agent.remote-id,1,1))),2), ":",
suffix (concat ("0", binary-to-ascii (16, 8, "", substring(option agent.remote-id,2,1))),2), ":",
suffix (concat ("0", binary-to-ascii (16, 8, "", substring(option agent.remote-id,3,1))),2), ":",
suffix (concat ("0", binary-to-ascii (16, 8, "", substring(option agent.remote-id,4,1))),2), ":",
suffix (concat ("0", binary-to-ascii (16, 8, "", substring(option agent.remote-id,5,1))),2)
);
}

# each static-IP subscriber gets a "static-sm" subclass entry like this
subclass "static-sm" "0a:00:3e:38:5a:e0";

# each static-IP subscriber gets a unique class with just their SM's MAC
class "alk-static" {
match if mysm = "0a:00:3e:38:5a:e0";
}

# BRAS/BNG Testing Subnets
shared-network brastest-default {
subnet 192.168.1.0 netmask 255.255.255.0 {
option routers 192.168.1.254;
}
pool {
# this is the static IP assignment for "alk-static"
# each static IP customer gets a pool definition like this one
allow members of "alk-static";
allow known-clients;
range 192.168.1.200;
}
pool {
# this is the default pool for dynamic customers
deny dynamic bootp clients; # DHCP is OK, BOOTP is not...
deny members of "static-sm"; # List of all static assignments (subclasses)
range 192.168.1.1 192.168.1.199;
}
}

ProTip: Forcing clean exit status on Linux (Bourne Shell) commands/scripts with “;:”

Often times in Linux (or BSD, Unix, etc) you want to trigger external commands, but don’t care if they succeed or fail. For example, in Debian, you can execute scripts and commands from /etc/network/interfaces using the [pre-]up and [post-]down directives. However, if something goes wrong and the command doesn’t exit cleanly, the interface action (ifup or ifdown) will fail. This can happen, for example, when adding a static route (ip route add ...) or alias address (ip addr add ...) that already exists ("RTNETLINK answers: File exists"). Another common scenario is using 'ethtool' to set interface parameters such as Tx/Rx queue sizes. If the ethtool command “fails” because the value is already set, that will cause the 'ifup' or 'ifdown' command to fail as well.

For example Continue reading →

SPAN port aggregation in FreeBSD

/etc/rc.conf:

ifconfig_em0="up promisc mtu 9212"
ifconfig_em1="up promisc mtu 9212"
cloned_interfaces="lagg0"
ifconfig_lagg0="laggproto loadbalance laggport em0 laggport em1 promisc monitor up mtu 9212"

usage: tcpdump -i lagg0 …

A utility to perform IPv4 & IPv6 prefix aggregation

I’m a fan of the aggregate program, which, in its own words, is used to “optimise a list of route prefixes to help make nice short filters”. However, it doesn’t work for IPv6! It was written by Joe Abley back in the 1990’s, and is useful in lots of scripts, but the lack of IPv6 support is starting to be an issue for me.

Using the IPy python module, I’ve created a similar tool that handles both IPv4 and IPv6 prefixes. Like the old tool, it can handle host addresses that don’t have a netmask specified, and can truncate the network portion to match the netmask if they are mismatched. I also added support for pre-1993 classful networking (since IOS and Quagga/Zebra still seem to leave the mask off such networks, two decades post-CIDR).

It doesn’t have all the features of the old aggregate command (i.e. –max-opt-lengh, etc), but it has all the features I used, and some that I had always wished for.

Continue reading →