When the kitchen lights stopped responding and the energy monitors went silent, I assumed it was a firmware issue on the devices themselves. Two hours of SSH automation and a surprising number of CLI quirks later, the culprit turned out to be a "friendly" enterprise WiFi feature that had been silently invalidating every device's password.
This is the full story — including every wrong turn, every automation failure, and the fix that brought 17 devices back online in under two minutes.
The Setup
The house spans four floors: cellar workshop, ground, first, and loft. The network runs three Ruckus R720 access points in Unleashed mode (firmware 200.15.6.212), plus a mix of legacy TP-Link EAP hardware and a handful of HH5A routers running OpenWRT as access points. The WiFi estate also includes a MR16 ceiling AP and a Workshop AP — both administratively offline for separate reasons.
The IoT fleet is substantial:
- Shelly EM energy monitors (bc:ff:4d:* OUI) — always-on, periodic heartbeat every 30–60 seconds
- TP-Link HS110 / KP115 smart plugs — ~10–30 second keepalive traffic
- Tasmota ESP8266 devices — default telemetry every 300 seconds, state-change-only transmission otherwise
- Custom ESP8266 PCB antenna light controllers — silent until someone flips a switch; can go hours without transmitting
- A Zigbee coordinator and various sensors, all fixed-location and physically inaccessible
None of these can be moved. None have easily accessible reset buttons. Rock-solid WiFi is a hard requirement.
The Channel Plan
Before investigating failures, worth documenting what's actually deployed. The channel plan is a deliberate 4-channel spread across the 2.4 GHz band:
| Floor | AP | Channel |
|---|---|---|
| Cellar/Workshop | Ruckus R720 | 1 |
| Ground | Ruckus R720 | 13 |
| First | Ruckus R720 | 6 |
| Loft | TP-Link / HH5A | 11 |
The Ruckus APs cover 5 GHz on separate non-overlapping channels (36/44/149 class), configured with VHT80 on supported hardware.
Automation First: Building the SSH Layer
Manual Ruckus CLI work is slow and error-prone. The first step was building a Python automation layer using paramiko.
Ruckus Unleashed has a peculiar authentication model: even after successful SSH authentication, the CLI requires a second username/password login at the shell level. This means the paramiko session needs to:
- Open an SSH connection
- Allocate a PTY and invoke a shell
- Wait for
Please login:— send username - Wait for
Password:— send password again - Detect whether this is the Unleashed master (
ruckus>prompt) or a subordinate AP (rkscli:prompt) - On the master, send
enableto enter privileged mode
The master presents a full hierarchical CLI with context-sensitive prompts. Every config level has its own prompt:
ruckus> # unprivileged
ruckus# # privileged (after 'enable')
ruckus(config)# # after 'config'
ruckus(config-wlan)# # after 'wlan <name>'
ruckus(config-wlan-acl)# # ACL sub-context
ruckus(config-aaa)# # AAA context
ruckus(config-ap)# # AP context
ruckus(config-ap-radio)# # radio sub-context
ruckus(config-sys)# # system context
The initial version of the script hardcoded cmd_prompt = "ruckus# " and used that as the sole wait target after every command. This worked until the first config command — after which the prompt changed to ruckus(config)# and the script hung indefinitely waiting for a prompt that would never come.
Fix: Replace the hardcoded prompt with a tuple of all possible master prompts, and pass all of them as alternatives to read_until():
MASTER_PROMPTS = (
"ruckus# ",
"ruckus(config)# ",
"ruckus(config-wlan)# ",
"ruckus(config-wlan-acl)# ",
"ruckus(config-aaa)# ",
"ruckus(config-ap)# ",
"ruckus(config-ap-radio)# ",
"ruckus(config-sys)# ",
"ruckus> ",
)
def read_until(chan, *prompts, timeout=30):
buf = ""
deadline = time.time() + timeout
while time.time() < deadline:
if chan.recv_ready():
chunk = chan.recv(4096).decode("utf-8", errors="replace")
buf += chunk
if "--More--" in buf:
chan.send(" ")
buf = buf.replace("--More--", "")
if any(p in buf for p in prompts):
return buf
else:
time.sleep(0.05)
raise TimeoutError(f"Timed out waiting for {prompts!r}. Got: {buf!r}")
Problem 1: Parallel Sessions Collide
The Ruckus master only allows one privileged session at a time. During investigation, two automation processes were launched in parallel to gather data faster — one to pull the active client list, one to check WLAN configuration.
The second process received:
A privileged user is already logged in.
Please try again later or use {force} option to login.
After this message, the master drops back to ruckus> (unprivileged). The script was not handling this case and timed out waiting for ruckus#.
Fix: All Ruckus CLI commands must run sequentially. The {force} option exists but using it forcibly disconnects the other session — not safe when the other session may be mid-commit.
Problem 2: Wrong WLAN Reference
Ruckus WLANs have both an internal numeric ID and a name. The CLI command to enter a WLAN context is:
wlan <name>
Early in the investigation, the command wlan 2 was sent, expecting it to select WLAN ID 2. Instead, Ruckus interpreted 2 as a name and created a brand-new WLAN called "2" with open authentication and no encryption. This new WLAN was immediately broadcast on all APs.
The mistake was compounded because the cleanup exit command saved the accidental WLAN to flash before the script timed out checking the result. The open WLAN was live for approximately 3 minutes before it was noticed and deleted:
config
no wlan 2
end
Lesson: Always use the WLAN name (wlan MiHA-2G), never a bare number. Verify show wlan output before committing changes.
Problem 3: show Command Discovery
The expected commands show wlan, show running-config, and show client all all returned:
% unrecognized or incomplete command
The correct approach was to run show ? to enumerate available subcommands. This revealed:
show current-active-clients all— lists all currently connected clientsshow current-active-clients mac <MAC>— documented to filter by MAC but actually returns the full list regardless
Note: show ? leaves the partial show command in the buffer, which then executes as an unrecognised command. This produces a harmless error message but is cosmetically annoying in script output.
Problem 4: The end Command Timeout
The end command in Ruckus CLI saves changes to running config and commits them to flash storage. For a simple WLAN change on MiHA-2G (2.4 GHz), this completed in about 8 seconds. For MiHA-5G (5 GHz), the flash write took over 15 seconds.
The initial timeout in the script was 15 seconds. MiHA-5G changes reliably failed with:
TimeoutError: Timed out waiting for ('ruckus# ', ...). Got: 'end\r\r\n'
The end echo had arrived, but the post-save ruckus# prompt had not yet appeared when the timeout fired.
Fix: Increase TIMEOUT from 15 to 30 seconds.
The Root Cause: Dynamic PSK "Friendly" Mode
With the automation layer working, the first serious diagnostic step was capturing the active client list and cross-referencing against known IoT devices. Seventeen devices that should have been connected were absent.
Checking recent authentication events in the Ruckus log revealed a pattern: repeated DEAUTH events for the same MAC addresses, cycling through connect → deauthenticate → connect attempts every few minutes. The error in the log:
Dynamic PSK authentication failed for client xx:xx:xx:xx:xx:xx on WLAN MiHA-2G
Ruckus Dynamic PSK ("friendly PSK") is an enterprise feature that generates a unique 62-character PSK per device on first connection. The device is enrolled with the shared passphrase, Ruckus registers its MAC, and thereafter that device must use its unique DPSK — the shared passphrase no longer works for it.
The problem: none of the IoT devices store the Ruckus-assigned DPSK. They store the static passphrase configured at the time they were set up. After Ruckus enrolled them and switched them to DPSK, every reconnection attempt using the static passphrase was silently rejected.
Devices that stay connected indefinitely (like TP-Link smart plugs with 10-second keepalives) never noticed. Devices that occasionally lose connectivity and need to re-associate — like the ESP8266 light controllers after a power cut, or any device that roams — were locked out permanently.
Fix: Disable Dynamic PSK on MiHA-2G:
config
wlan MiHA-2G
no dynamic-psk
end
Within two minutes, 17 devices reconnected. The Ruckus event log showed a flood of successful associations starting at 15:36:57.
Problem 5: Inactivity Timeout Too Aggressive
With DPSK resolved, a second problem emerged in real time during the session.
Ruckus has a client inactivity timeout: if a client sends no frames for N minutes, Ruckus deauthenticates it. The default is 5 minutes.
At 15:36, kitchen-lights-main-5994 appeared in the active client list. At 16:20 — just 44 minutes later — it was gone. The light controller is a state-change-only device: it only transmits when a light state changes. In a house where no lights were manually operated during that window, the device was completely silent.
A full audit of all IoT device transmit patterns:
| Device type | Worst-case silence | Risk at 5 min |
|---|---|---|
| TP-Link HS110/KP115 | ~60 seconds | None |
| Shelly EM | ~60 seconds | None |
| Tasmota (TelePeriod 300) | 300 seconds | At the boundary |
| ESP8266 light controllers | Hours (state-change only) | Certain disconnect |
Tasmota's default 300-second telemetry interval is exactly at the 5-minute boundary. Any clock drift or network delay could push it over.
Fix: Set inactivity timeout to 60 minutes on both WLANs:
config
wlan MiHA-2G
inactivity-timeout 60
end
config
wlan MiHA-5G
inactivity-timeout 60
end
Secondary recommendation: set TelePeriod 120 on Tasmota devices to give 2.5× margin against any future timeout changes.
Outstanding Issues
Not everything was resolved in this session:
xhome-kitchen-sinklight still offline. This ESP8266 device (MAC 2c:f4:32:bf:c1:15) did not reconnect after DPSK was disabled, even after a full power cycle of the circuit. The last authentication failure in the Ruckus log was at 15:33:14, before DPSK was disabled. No new attempts visible. Possible causes: (a) device has a Ruckus-assigned DPSK stored in its firmware rather than the static passphrase — effectively bricked until factory reset; (b) signal coverage gap on the ground floor following the Dining HH5A going offline; (c) the device is configured to a different SSID.
Ground floor coverage gap. The Dining area HH5A (10.140.2.11) is offline. The ground floor Ruckus AP covers the main areas but may not reach all corners with sufficient signal.
Workshop AP radios disabled. The Workshop Ruckus AP has all WiFi radios administratively disabled. The cellar and workshop areas have no wireless coverage.
MR16 Hall AP radios disabled post-firmware upgrade to 25.12.2.
MiHA-5G DPSK not yet disabled. Only MiHA-2G was fixed in this session. Any 5 GHz IoT devices that went through DPSK enrollment face the same issue.
Takeaways
Dynamic PSK "friendly" mode is a footgun for static IoT deployments. It's designed for BYOD environments where devices are enrolled by users. In a home automation context where devices are never user-managed, DPSK creates a silent one-way door: devices connect once, get enrolled, and then can never reconnect after any interruption.
The 5-minute inactivity timeout is incompatible with state-change-only IoT. Light controllers, sensors, and similar devices may be silent for hours. A 60-minute timeout is a reasonable baseline; some installations may need it even higher.
Ruckus CLI automation with paramiko requires careful prompt handling. The double-authentication model and context-sensitive prompts make it easy to write a script that works for simple cases and silently fails on config changes. The tuple-of-prompts approach handles all context levels correctly.
Parallel SSH sessions cause hard failures on Ruckus Unleashed. The privileged session lock is enforced strictly. Sequential execution is required.
Always test WLAN changes with a named reference. wlan 2 ≠ wlan MiHA-2G. The former creates a new WLAN; the latter enters the existing one.
Investigation conducted 2 May 2026. Ruckus Unleashed firmware 200.15.6.212 build 27. Three R720 APs in Unleashed cluster.
No comments:
Post a Comment