Deep Dive
19 February 2021 / /
High Availability L2TP LNS Steering with FreeRADIUS and ExaBGP
We are in the final stages of coming onboard with Zen Internet for wholesale L2TP services. Yesterday their technical delivery team moved on to testing our RADIUS integration. We were a little nervous that this would not work first time as most L2TP wholesale providers just expect their customers to interoperate perfectly with the appropriate RADIUS responses. The tests were successful, thanks to plenty of careful preparation.
Given our extensive experience using FreeRADIUS we were fairly confident it would be part of our solution so started there. FreeRADIUS powers the access control on our own management networks, and authenticates multi-network 3G/4G wholesale SIMs onto the partner mobile carrier networks for Key IoT. We just needed to work out how to make it sing the right tune for L2TP. We didn’t have a large vendor’s stack to fall back on, so started from the RFCs and other implementations’ documentation. We hoped that this would steer us (excuse the pun) to build the right configuration.
Our requirements when designing our L2TP tunnel-steering RADIUS configuration were as follows:
- respond to RADIUS requests for multiple realms
- steer realms to be L2TP tunneled to one or more appropriate LNSs
- per-realm and/or per-LNS tunnel secrets to support multi-tenancy
- load-balance incoming sessions across the realm’s LNSs
- automated deployment of nodes’ configurations using SaltStack
- use BGP to advertise RADIUS server IPs to the wholesale/carrier networks
- nodes should anycast their RADIUS service IP addresses into our IGP
- anycast addresses should only be announced if the service is running; withdraw if the RADIUS serving process is unresponsive
Salt Pillar Data
We define all the realms, LNSs and L2TP tunnel secrets in pillar data so that it’s available across all the relevant minions. The structure we use is like this (shown below in Salt’s standard SLS/YAML format):
direct: # pillar data for XDSL realms and their LNS steering
realms: # mapping from realm names (user@realm) to realm data
realm1.example.com:
lns: # mapping of LNS IP address to LNS L2TP shared secret
192.0.2.1: hunter2
realm2.example.com:
lns: # steering to randomly-ordered set of multiple LNSs
192.0.2.2: hunter2
192.0.2.3: hunter2
192.0.2.4: hunter2
192.0.2.5: hunter2
healthcheck: # fake realm for automated healthchecks
lns:
127.0.0.1: success
redirect: # user goes to specified realm's LNSs
test@other.example.com: realm1.example.com
Based on our experience while helping a customer migrate the infrastructure we added functionality to redirect individual users’ sessions to other realm’s LNSs. This can be used to help customers either, for example, during a realm swing; or it could be used to redirect a problematic session to a debugging LNS to determine why a particular session is failing to establish.
FreeRADIUS Site
This Jinja template is deployed with a file.managed state by Salt as an available site for FreeRADIUS.
{% set direct = salt['defaults.merge'](salt['pillar.get']('direct',{}), salt['grains.get']('direct',{})) %}
server direct {
{# elided "listen" and "client" sections #}
authorize {
if ("%{request:User-Name}" =~ /^(.+)@(.+)/) {
update {
request:Realm := "%{2}"
}
}
{% if direct.get('redirects',{}) %}
switch &request.User-Name {
{% for redirect, destination in direct.get('redirects',{}).items() %}
case "{{ redirect }}" {
update {
request:Realm := "{{ destination }}"
}
}
{% endfor %}
}
{% endif %}
switch &request:Realm {
{% for realm, realmdata in direct.get('realms',{}).items() %}
case "{{ realm }}" {
{% set lns = realmdata.get('lns',{}).items()|list %}
{% for offset in range(lns|length) %}
{% if loop.first and not loop.last %}
switch "%{rand: {{ lns|length }}}" {
{% endif %}
{% if not ( loop.first and loop.last ) %}
case "{{ offset }}" {
{% endif %}
update {
{% for i in range(lns|length) %}
{% set j = (i+offset) % (lns|length) %}
reply:Tunnel-Server-Endpoint:{{i}} = "{{ lns[j][0] }}"
reply:Tunnel-Password:{{i}} = "{{ lns[j][1] }}"
reply:Tunnel-Type:{{i}} = L2TP
reply:Tunnel-Medium-Type:{{i}} = IP
{% endfor %}
control:Auth-Type = "Accept"
}
{% if not ( loop.first and loop.last ) %}
}
{% endif %}
{% if loop.last and not loop.first %}
}
{% endif %}
{% endfor %}
ok
}
{% endfor %}
}
}
}
The above template ends up generating a configuration file which is fairly verbose (for additional load-balancing functionality), but is functionally similar to this:
server direct {
client healthcheck {
ipv4addr = 127.0.0.1/32
shortname = healthcheck
secret = healthcheck
}
# client wholesale { ... etc ... }
authorize {
if ("%{request:User-Name}" =~ /^(.+)@(.+)/) {
update {
request:Realm := "%{2}"
}
}
switch &request.User-Name {
case "test@other.example.com" {
update {
request:Realm := "realm1.example.com"
}
}
}
switch &request:Realm {
case "healthcheck" {
update {
reply:Tunnel-Server-Endpoint:0 = "127.0.0.1"
reply:Tunnel-Password:0 = "success"
reply:Tunnel-Type:0 = L2TP
reply:Tunnel-Medium-Type:0 = IP
control:Auth-Type = "Accept"
}
ok
}
case "realm1.example.com" {
update {
reply:Tunnel-Server-Endpoint:0 = "192.0.2.2"
reply:Tunnel-Password:0 = "hunter2"
reply:Tunnel-Type:0 = L2TP
reply:Tunnel-Medium-Type:0 = IP
reply:Tunnel-Server-Endpoint:1 = "192.0.2.3"
reply:Tunnel-Password:1 = "hunter2"
reply:Tunnel-Type:1 = L2TP
reply:Tunnel-Medium-Type:1 = IP
# reply:Tunnel-Server-Endpoint:2 = "192.0.2.4" ... etc ...
control:Auth-Type = "Accept"
}
ok
}
case "realm2.example.com" {
# ... etc ...
}
}
}
}
ExaBGP Health Check
To periodically check that FreeRADIUS is running and announce or withdraw the relevant loopback addresses via BGP we created a small shell script. This can be installed as /usr/local/sbin/radius-steering-healthcheck for example.
We set the addrs variable to be the list of non-loopback IP addresses of the lo interface. This is how we define the node’s /32 address(es) to advertise into BGP if FreeRADIUS is running correctly.
The username can be anything@healthcheck (so long as the realm and a RADIUS secret match)
#!/bin/bash
addrs=`ip -4 addr show dev lo scope global | grep inet | awk '{ print $2 }'`
while true
do
if radtest "healthcheck@healthcheck" "healthcheck" "127.0.0.1" "1" "healthcheck" > /dev/null
then
for addr in $addrs
do
echo "announce route $addr next-hop self"
done
else
for addr in $addrs
do
echo "withdraw route $addr next-hop self"
done
fi
sleep 15
done
To use the healthcheck with ExaBGP 4.0 we added the following section to exabgp.conf:
process healthcheck {
run "/usr/local/sbin/radius-steering-healthcheck";
encoder text;
}
Bonus Content: Virtual LNSs with VyOS under Xen
While performing our own tests, which included testing with virtual LNSs running as virtualised network functions, we found a problem setting the MTU on interfaces in VyOS 1.3. Another user had posted on the VyOS forum with the same problem eight months earlier. Just like them, our virtualisation stack is running Xen.
What puzzled us is that on a standard Debian Linux virtual machine running on the same NFV cluster we had no problems setting the MTU to support jumbo frames:
root@buster:~# ls -l /sys/class/net/eth0/device/driver
lrwxrwxrwx 1 root root 0 Feb 19 05:43 /sys/class/net/eth0/device/driver -> ../../bus/xen/drivers/vif
root@buster:~# ip link set dev eth0 mtu 2000
root@buster:~# ip link set dev eth0 mtu 9000
root@buster:~# ip link set dev eth0 mtu 1500
And yet on VyOS, itself a Debian-derived distribution, we could not increase MTU beyond 1500:
vyos@equuleus:~$ ls -l /sys/class/net/eth0/device/driver
lrwxrwxrwx 1 root root 0 Feb 19 05:45 /sys/class/net/eth0/device/driver -> ../../bus/xen/drivers/vif
vyos@equuleus:~$ configure
[edit]
vyos@equuleus# set interfaces ethernet eth0 mtu 9000
[edit]
[ interfaces ethernet eth0 ]
VyOS had an issue completing a command.
We are sorry that you encountered a problem while using VyOS.
There are a few things you can do to help us (and yourself):
- Make sure you are running the latest version of the code available at
https://downloads.vyos.io/rolling/current/amd64/vyos-rolling-latest.iso
- Consult the forum to see how to handle this issue
https://forum.vyos.io
- Join our community on slack where our users exchange help and advice
https://vyos.slack.com
When reporting problems, please include as much information as possible:
- do not obfuscate any data (feel free to contact us privately if your
business policy requires it)
- and include all the information presented below
Report Time: 2021-02-19 05:46:48
Image Version: VyOS 1.3-rolling-202012251712
Release Train: equuleus
Built by: autobuild@vyos.net
Built on: Fri 25 Dec 2020 17:12 UTC
Build UUID: debe2b8f-0337-4bbf-8d5a-6bfd80111dd3
Build Commit ID: 6bf09791f97fae
Architecture: x86_64
Boot via: installed image
System type: Xen PV guest
Hardware vendor: Unknown
Hardware model: Unknown
Hardware S/N: Unknown
Hardware UUID: Unknown
OSError: [Errno 22] Invalid argument
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/libexec/vyos/conf_mode/interfaces-ethernet.py", line 102, in <module>
apply(c)
File "/usr/libexec/vyos/conf_mode/interfaces-ethernet.py", line 94, in apply
e.update(ethernet)
File "/usr/lib/python3/dist-packages/vyos/ifconfig/ethernet.py", line 298, in update
super().update(config)
File "/usr/lib/python3/dist-packages/vyos/ifconfig/interface.py", line 1264, in update
self.set_mtu(config.get('mtu'))
File "/usr/lib/python3/dist-packages/vyos/ifconfig/interface.py", line 361, in set_mtu
return self.set_interface('mtu', mtu)
File "/usr/lib/python3/dist-packages/vyos/ifconfig/control.py", line 182, in set_interface
return self._set_sysfs(self.config, name, value)
File "/usr/lib/python3/dist-packages/vyos/ifconfig/control.py", line 166, in _set_sysfs
self._sysfs_set[name]['location'].format(**config), value)
File "/usr/lib/python3/dist-packages/vyos/ifconfig/control.py", line 132, in _write_sysfs
f.write(str(value))
OSError: [Errno 22] Invalid argument
[[interfaces ethernet eth0]] failed
Commit failed
The problem did not seem to be one caused directly by VyOS, as we could not use the standard Linux tools to make the same change:
vyos@equuleus:~$ sudo ip link set dev eth0 mtu 9000
RTNETLINK answers: Invalid argument
vyos@equuleus:~$ sudo ifconfig eth0 mtu 9000
SIOCSIFMTU: Invalid argument
vyos@equuleus:~$ echo 9000 | sudo tee /sys/class/net/eth0/mtu > /dev/null
tee: /sys/class/net/eth0/mtu: Invalid argument
Delving into the Linux kernel we found that Xen’s xen-netback driver has a particular quirk: if “scatter-gather” offloading (can_sg) is disabled then xenvif_change_mtu will limit the maximum MTU to 1500. Checking a VyOS virtual LNS we could see that, by default, this offloading is disabled:
vyos@equuleus:~$ sudo ethtool -k eth0
Features for eth0:
rx-checksumming: on [fixed]
tx-checksumming: on
tx-checksum-ipv4: on [fixed]
tx-checksum-ip-generic: off [fixed]
tx-checksum-ipv6: on
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: off [fixed]
scatter-gather: off
tx-scatter-gather: off
tx-scatter-gather-fraglist: off [fixed]
This was easily fixed by enabling sg offload in VyOS as follows:
set interface ethernet eth0 offload sg
set interface ethernet eth0 mtu 9000
We reported our findings to the VyOS project in a bug report, and also followed-up on the thread from the other user having the same problems in a similar environment.