Most smart home security write-ups pick one device and pull it apart. I wanted the opposite: point a full assessment at an entire smart home lab at once (router, NAS, heating controller, smart meter bridge, everything on the lab network) and see what falls out. The hub that controls all 37 paired devices accepts commands from anyone on the network. No password, no token, nothing. Motion detector as a presence oracle, thermostat setpoints and blinds writable with one request.

To be clear up front: this is a smart home lab, a standing environment from my master's thesis project that I test with the owner's permission, on an isolated network. Everything below was tested there and nowhere else.

But the findings are only half the point. This assessment didn't run by hand — it ran on a scaffold I designed during my master's thesis and a harness I built afterwards, an AI-orchestrated pipeline where the model does the operator work inside structure I designed for it. The bugs are the output. This post is mostly about the machine that produced them.

The scaffold: agentic testing as a file system

The structure comes from the master's thesis platform (MWP) that this lab's assessments run on, and I've kept evolving it since. The core design decision: an AI-driven assessment is a layered file system that any agent context can be dropped into, does its work in, and gets replaced from. The chat is just the interface.

{Category}/{Device}/
├── CLAUDE.md           # layer 0: identity — rules, tool inventory, learned quirks
├── CONTEXT.md          # layer 1: task routing — which stage, which entry point
├── _config/            # scope.md, device-profile.md
├── stages/01..06/      # the assessment pipeline, each with its own CONTEXT
└── notes/ leads/ gadgets/ findings/ reports/

Three design choices make it hold together.

The first is the note funnel, and it works as a validation gradient. Raw observations land in notes/, anything worth chasing graduates to leads/, reusable techniques become gadgets/, validated bugs with a PoC go to findings/, and reports/ is client-ready output. An observation is not a finding until it has survived investigation and carries a proof; without that, an eager model happily files "port open" next to "authenticated RCE" as equals.

Memory has to survive compaction, because long agentic sessions lose context — that's just physics of the context window. So progress persists in files: a session-state.md checkpoint gets written before compaction, and a Redis-backed RAG store (search, ingest, finding, gadget commands) is shared across every device assessment. When I find a parser quirk on one CCU, the platform recalls it on the next one. The scaffold learns.

Then autonomy. Saying "keep testing" switches the pipeline into unattended operation: no questions, reasonable defaults, thorough notes, state checkpointed so the run survives overnight. Between stages there are review gates, and findings get mapped to OWASP ISTG and MITRE ATT&CK ICS so the report is comprehensible to someone who isn't me.

What it found: the proof the pipeline works

The heaviest result came from the HomeMatic CCU — the RaspberryMatic hub that runs the whole radio stack. Its two XML-RPC interfaces (ports 2001 for classic BidCoS, 2010 for HmIP) exist so the CCU can talk to wired counterparts, and neither asks who is talking. Unauthenticated XML-RPC requests to those interfaces read out any device's full parameter set (temperatures, humidity, valve and window states), and a second call form writes values: thermostat setpoints, blind positions, switch states.

I enumerated everything reachable: 37 paired devices. Six smoke detectors with readable alarm state, ten wall and radiator thermostats with writable setpoints, two floor heating controllers (twelve zones each), three blind actuators that obey, a motion detector whose flag doubles as a presence oracle, three plug switches, sensors, a wall button panel. All of it from any host on the LAN.

This is a known design flaw (CVE-2019-9583), not a fresh 0day, but "known" and "still the factory default in 2026" are different things. CVSS 9.6 in this context: adjacent network, no privileges, no interaction, full CIA on every device. OWASP ISTG maps it to ISTG-DES-AUTHZ-001, missing authentication for a critical function. The embedded XML-RPC parser also turned out picky about value encoding, which cost me a few minutes and a gadget note.

The supporting cast: the QNAP NAS serves phpinfo() as its default web page (version, paths, and the detail that the web user sits in the administrators group); the AMS energy bridge (the thing that forwards the Norwegian smart meter's HAN port to the LAN) exposes unauthenticated JSON endpoints for live power, per-phase data, and daily/monthly history, which is a household presence indicator wearing a smart-home badge; and the Fritz!Box login_sid.lua endpoint hands out every configured username, no auth. Exact payloads and endpoints are deliberately left out of this post; they live in the lab's gadget files.

The harness: doctrine compiled into the model

Running the thesis scaffold was also what showed me its ceiling. The scaffold constrains the session — what it may touch, where it may write, when it may run unattended, and when it needs to stop and ask. But every session still started from a default assistant identity that had to be persuaded, prompt by prompt, to behave like an operator. So I built the thing I actually wanted: PenTestCLI, a fork of the open-sourced Claude Code with a red team operator system prompt compiled into the build. The default coding-assistant identity never ships; the operator one does.

Two rules in that doctrine do most of the work.

Impact first. A finding without a reproducible proof of concept is a lead, not a finding, and you score only the impact the PoC demonstrates. This is the note funnel turned into software: the model can't graduate an observation to a finding because its identity forbids it, and a markdown file never had that power.

Closure discipline. Every candidate ends in exactly one state: confirmed (working PoC), ruled out (naming the specific control that stops it), or open (naming what would resolve it). "I moved on" is not a state. It kills the classic LLM failure mode of an investigation quietly trailing off, with only the survivors making it into the report.

Both come from one design idea. The file-based scaffold is external process structure: stage gates, the note funnel, scope documents, checkpoint files. The compiled identity is internal doctrine — the methodology loop (recon, map, identify, exploit, validate, chain, report) as the model's native vocabulary. The result behaves less like an assistant following instructions and more like someone who's read the methodology and can't un-read it.

The Josef_lab environment doubled as the test bed: every quirk and gadget from this assessment feeds back into the harness's memory, and every session sharpens the doctrine.

Where it's heading

The honest threat-model question for everything above is "what about someone not already inside the LAN?" The current goal on the board is an external path: unauthenticated access to Home Assistant, then a pivot toward the CCU. No-interaction server-side flaws are hard, so the lab is doing the patient work: AFL++ chewing on the CCU's interfaces and Atheris on Home Assistant, crash triage when something lands, everything flowing through the same funnel.

If you run this gear: segment the IoT VLAN, scope who can reach the CCU's radio interfaces, delete the phpinfo page, and decide whether you're okay with your energy bridge being readable. The full notes, gadget files, and findings live in the lab repo — this post is the tour, not the paperwork.

Update (October 2026): the thesis has since moved on from this lab to its next environment — a power transformer substation. Same platform, same funnel, new domain.