Jeff Sharkey

I manage a handful of networks in remote locations, and IPMI has proven to be an invaluable debugging and recovery tool, especially when fixing issues that would have required jumping on an airplane.

Historically, I've been relying on a lightweight secondary switch (typically a faithful WRT54GL) to provide an OpenVPN tunnel into a completely isolated LAN that hosts core infrastructure like IPMI and PDU management. But in a recent deployment I was limited to a single public-facing IPv4 address, and I really wanted to avoid routing all remote access through a single point of failure.

To work around this limitation, I started looking at building a completely out-of-band management interface using a WWAN cellular modem. This has the additional benefit of being completely redundant from the normal wired ISP. I've been using Project Fi (now Google Fi) for a few years now, and they have a neat feature that allows you to add up to 10 data-only SIM cards at no extra cost, other than paying for the data that they use at $10/GB.

For hardware, I narrowed in on the MikroTik RBM33G, which is packed with a ton of useful features which make it incredibly versatile. Here are just some of the ideas I've been brainstorming:

• First miniPCIe port for WWAN card
• Second miniPCIe port for local Wi-Fi monitoring, for automated alerting when APs wedge before getting support calls from humans
• RS232 port for managing things like stacked network switches
• USB 3.0 port, which has endless possibilities, such as a serial port hub if you have other hardware to manage
• Easily powered via a passive PoE injector, and provides 3 built-in Ethernet LAN ports, which is enough to support most smaller deployments

First attempt: RouterOS

The idea sounded promising, so I picked up the RBM33G with a R11e-LTE-US and started playing around with RouterOS. After updating to the latest firmware, it connected to the Google Fi APN (h2g2) pretty easily, but it needed these commands to help it along:

/system routerboard usb set type=mini-PCIe
/system routerboard settings set init-delay=5s

It looks like these Google Fi data-only SIMs are using T-Mobile under the hood, since the carrier network has the tell-tale 464XLAT design that routes you through RFC 1918 address space before reaching the Internet.

I hesitated to enable IPv6, knowing MikroTik's troubled past in that area, and I also wasn't impressed with the security options of the built-in OpenVPN server/client, as it only supports MD5 and SHA-1, both of which are considered obsolete.

Second attempt: OpenWRT

At this point I gave up on RouterOS, and found that OpenWRT has solid support for the RBM33G. After downloading and flashing everything, getting a real Linux box was a breath of fresh air, with all sorts of wonderful debugging tools at my fingertips.

The biggest pain was that the R11e-LTE-US was now really unhappy, and it required lots of creative hand-holding to start working, and even then it was flaky. I gave up on the R11e-LTE-US entirely and switched over to a SIMCom SIM7100A, which I found to be much more reliable.

Here's the set of repeatable steps that I'm using for configuring each new router before deploying into production:

1. Activate the Google Fi SIM card online
2. Install SIM and modem into the router, and attach antenna
3. Follow OpenWRT flashing instructions and install using sysupgrade
4. Disable password logins in /etc/config/dropbear
5. Install SSH keys in /etc/dropbear/authorized_keys
6. Install basic packages:

opkg update
opkg install usb-modeswitch kmod-mii kmod-usb-net kmod-usb-wdm kmod-usb-net-qmi-wwan uqmi
opkg install wireguard

7. Add WWAN interface to /etc/config/network:

config interface 'wwan'
	option ifname 'wwan0'
	option proto 'qmi'
	option device '/dev/cdc-wdm0'
	option apn 'h2g2'
	option pdptype 'ipv4'

8. Reboot the device to kick everything into place, and then confirm that I can reach the Internet via WWAN

Remote access

For the remote access tunnel into the router, I've been using WireGuard, which I've found to be a really nice VPN solution, both from a security and performance standpoint.

Here's an example of a typical network topology that I'm using:

• 192.168.100.0/24 used for WireGuard internals
• 192.168.101.0/24 network at remote site "Red"
• 192.168.102.0/24 network at remote site "Blue"

Here's a typical WireGuard configuration on the server side:

[Interface]
ListenPort = [server-port]
PrivateKey = [server-private-key]

[Peer]
PublicKey = [red-public-key]
AllowedIPs = 192.168.101.0/24
[Peer]
PublicKey = [blue-public-key]
AllowedIPs = 192.168.102.0/24

Along with some routing rules so the server can reach out into the remote client networks:

ip route add 192.168.101.0/24 dev wg0
ip route add 192.168.102.0/24 dev wg0

And here's a typical WireGuard configuration on the client side:

config interface 'lan'
...
	option ipaddr '192.168.101.1'

config interface 'wg0'
	option proto 'wireguard'
	option listen_port '[server-port]'
	list addresses '192.168.100.101/32'
	option private_key '[red-private-key]'

config wireguard_wg0
	option public_key '[server-public-key]'
	option route_allowed_ips '1'
	list allowed_ips '192.168.100.0/24'
	option endpoint_host '[server-host]'
	option endpoint_port '[server-port]'
	option persistent_keepalive '25'

While I'm configuring the router, I like to drop the wan and wan6 interfaces, and reconfigure port 0 so I can use all three physical Ethernet ports as LAN:

config switch_vlan
...
	option ports '0 1 2 3 4 6t'

And there you have it: you should now be able to reach out from the server to any devices on the remote client networks, such as an IPMI host at 192.168.101.99.

I've been using a persistent_keepalive value of 25 seconds to keep the tunnel alive through the 464XLAT carrier-grade NAT. I chose 25 seconds since I wasn't worried about power usage, but if you wanted to tune that value further, I wrote a tool to empirically derive NAT timeouts for carrier networks.

Conclusions

The first router has been running solid for a few months now, and I'm deploying a few more into production shortly. Here's the final parts list I'm using:

Item	Price
MikroTik RBM33G board	$44.00
MikroTik CA/433U case	$18.75
SIMCom SIM7100A	$41.79
RP SMA Female to uFL pigtail	$1.03
7dBi external antenna with 3m cable	$4.99
Total price	$110.56

The idle keep-alive traffic averages 0.5MB/day, which comes out to a very affordable $0.15/month based on the current Google Fi billing rates.

Overall I'm really happy with the versatility of the MikroTik hardware, combined with the reliability of OpenWRT and the inexpensive Google Fi data-only SIM.