Building a Triggerable Home Lab Health Check with Python and systemd

The Claude Code sessions on my laptop dropped out mid-conversation. The internet felt fine — pages loaded, DNS resolved — but I couldn't be certain whether something had shifted on the LAN. What I wanted was a single command that would sweep through everything: public internet reachability, DNS resolution, gateway, every Incus container with a .lan address, and a handful of remote hosts. One invocation, one exit code.

What I discovered while building it made the effort worthwhile almost immediately.

The DHCP drift problem

My home lab includes a TP-Link Kasa HS110 smart plug (running on python-kasa) that controls mains power to a Proxmox node with no IPMI — it's the only remote kill switch available. That plug's IP was documented everywhere as 10.140.1.235. The health check tried it and got silence.

A quick nmap sweep by MAC address found it immediately at 10.140.1.236:

sudo nmap -sn 10.140.0.0/22 | grep -B2 D8:47:32:A3:FF:CA

Nmap scan report for HS110.lan (10.140.1.236)
Host is up (0.075s latency).
MAC Address: D8:47:32:A3:FF:CA (TP-Link Technologies)

The DHCP lease had quietly shifted by one. The plug was on, PVE2 was drawing 126 W and happily running — I just wouldn't have been able to reach it from any documented procedure. The lesson is obvious in hindsight: for anything critical, find it by MAC, not by IP. The health check script now tries the known IP first and falls back to an nmap sweep if it doesn't respond, reporting the new address so you can update your docs.

Adding missing .lan entries

While auditing the LAN, I found two services without .lan entries: Gitea (running in an LXC container on a second Proxmox node at 10.140.3.116:3000) and the Xenon Proxmox node itself (at 10.140.3.82:8006). Both get Caddy reverse proxy entries — Xenon needs a tls_insecure_skip_verify because Proxmox uses a self-signed certificate:

# /etc/caddy/sites-enabled/gitea.caddy
gitea.lan {
    tls internal
    reverse_proxy http://10.140.3.116:3000
}

# /etc/caddy/sites-enabled/xenon.caddy
xenon.lan {
    tls internal
    reverse_proxy https://10.140.3.82:8006 {
        transport http {
            tls_insecure_skip_verify
        }
    }
}

sudo caddy validate --config /etc/caddy/Caddyfile && sudo systemctl reload caddy

The DNS priority conflict

Here's where it got interesting. After reloading Caddy, curl https://gitea.lan/ returned a connection refused — but not because of Caddy. Running resolvectl query gitea.lan showed it resolving to 10.140.3.116 directly rather than 127.0.0.1 (which is what the dnsmasq wildcard should return).

The setup uses dnsmasq at 127.0.0.2 with address=/.lan/127.0.0.1 to catch all .lan names and send them to Caddy. systemd-resolved is configured in /etc/systemd/resolved.conf.d/lan.conf to route .lan to that dnsmasq instance:

[Resolve]
DNS=127.0.0.2
Domains=~lan

The problem: the wired ethernet DHCP lease (from the router at 10.140.2.6) also pushes domain=lan, which means systemd-resolved registers the router as a per-link routing DNS for .lan as well. Since the router has a genuine DHCP record for gitea.lan (it knows about the Gitea container's lease), it answers first — and systemd-resolved never falls back to dnsmasq.

Setting dns-priority=100 on the NetworkManager connection should tell systemd-resolved to prefer the global server, but the change didn't propagate cleanly in this session. The reliable workaround is an /etc/hosts entry:

printf '127.0.0.1 gitea.lan\n127.0.0.1 xenon.lan\n' | sudo tee -a /etc/hosts

Inelegant, but unambiguous. getent hosts gitea.lan immediately returned 127.0.0.1 and Caddy took over correctly. The underlying dns-priority issue is worth a proper investigation another day — the nmcli setting is:

sudo nmcli con mod "Wired connection 3" ipv4.dns-priority 100
sudo nmcli con mod "MiHA-2G 2" ipv4.dns-priority 100

The health check script

The script lives at agents/lan_health.py and covers four categories: internet (ICMP to 1.1.1.1 and 8.8.8.8, pass if either responds), DNS resolution, LAN host pings, and HTTPS checks against every .lan service. The HS110 section is the most useful part:

HS110_MAC = "D8:47:32:A3:FF:CA"
HS110_IP  = "10.140.1.236"   # DHCP — may drift

def find_hs110():
    r = subprocess.run(
        ["sudo", "nmap", "-sn", "--open", "10.140.1.0/24"],
        capture_output=True, text=True, timeout=30
    )
    lines = r.stdout.splitlines()
    for i, line in enumerate(lines):
        if HS110_MAC.upper() in line.upper():
            for j in range(max(0, i-3), i):
                if "report for" in lines[j]:
                    return lines[j].split()[-1].strip("()")
    return None

If the known IP doesn't respond, find_hs110() runs the sweep and returns the new IP (or None if the plug genuinely isn't on the network). Output is colour-coded; exit code is non-zero on any failure, which makes it composable with scripts or monitoring.

The systemd service

The whole point is to trigger this on demand rather than poll continuously. A oneshot unit is the right shape:

# /etc/systemd/system/lan-health.service
[Unit]
Description=Home LAN health check
After=network-online.target
Wants=network-online.target

[Service]
Type=oneshot
ExecStart=/home/user/claude/.venv/bin/python3 /home/user/claude/agents/lan_health.py
User=user
StandardOutput=journal+console
StandardError=journal+console

sudo systemctl daemon-reload
systemctl start lan-health.service
journalctl -u lan-health.service -n 50 --no-pager

No persistent daemon, no polling — it runs when asked, logs to the journal, and exits. StandardOutput=journal+console means you see the output immediately if you're watching, and journalctl -u lan-health.service gives you the last run any time you want it.

First invocation caught the HS110 drift, three services returning 502 (backends stopped, proxies still up), and confirmed everything else was healthy. Not a bad return for an afternoon's work.

References

• nmap — network discovery; -sn ping scan, MAC address lookup in sweep output
• Caddy v2 — automatic HTTPS reverse proxy; tls internal for self-signed .lan certs, tls_insecure_skip_verify for upstream self-signed
• dnsmasq — lightweight DNS/DHCP; address=/.lan/127.0.0.1 wildcard used here
• systemd-resolved — stub resolver; Domains=~lan routing domain config
• resolvectl(1) — resolvectl query for diagnosing which DNS server answered
• NetworkManager / nmcli — ipv4.dns-priority for per-link DNS priority
• systemd.service(5) — Type=oneshot for run-and-exit services
• python-kasa — TP-Link Kasa smart device control (used separately to read HS110 wattage)
• Python stdlib: subprocess, socket, urllib.request, ssl — all used in the health check script

Ta ta for now, Matt