As you know, I’ve been reading Why We Still Suck at Resilience by Adrian Hornsby (yes, I know, it’s like n-th post that starts like that, but what can I say - I’m a slow, thoughtful reader and it gives me ideas), and I read about “organisational learning” vs “learning organisation”. tldr; would be - if the companies pays attention to incidents, and after each one let’s say they add a control, or add a new verification step to their CI pipeline, then we can say this is “organisational learning” - an organisation learned something and changed their behaviour based on that. It is a good step, not every organisation has it. The problem with it though, is that they might get into accumulating documentation, accumulating validation steps and extra checks, runbooks of runbooks. They have a “how to do X” page about all kinds of issues they have encountered. But they haven’t built the knowledge and understanding of how to deal with “novel” incidents.
Does that sound familiar? Let’s bring this to agentic world - you’ve created an agent, and every time it does something wrong you tell it to not do it the next time. You add that to your SOMETHING.md, you share it with your team, so they also benefit from that “rule”. You might even automate that part, like I did for my local agent, where I created a hook that is firing every time a single conversation is taking “too many turns” and is reflecting on the conversation itself, and on the things that I highlighted as “issues” and on how could they have been avoided in the first place, then let’s me pick and choose what to store in memory for the future.
That works for a while, but every now and then your agent ignores the rules, deviates from them, or encounters a “slightly” different flavor of what you wrote, where you think like “well, you could have inferred that”. You are frustrated, you read about it, and you learn that you can’t fill the context forever with rules. You also hear that the agents are not that good at “novel” incidents (and I call “incident” anything that didn’t go as originally planned). Well, of course they aren’t. Because they are, at best, applying “organisational learning”.
Let’s read further, and see how can we go from “organisational learning” to a “learning organisation”. For that, a company would need to think more “meta”. They would need to ask themselves questions like “what patterns are we seeing across the incidents?”, “what does it tell us about the gap between how we thought the system works, and how it actually works?”, “how does this affect other teams work?”. Point being, a learning organisation uses incidents to generalize across a larger scope and learn on a larger scope.
Next, Adrian brings in a classic categorization of learning types (it goes back to Argyris and Schön, organizational-learning people from the 70s):
“Single-loop” - add yet-another check based on the incident we observed
“Double-loop” - “questions the assumptions and conditions within which single-loop learning operates” - why do we keep adding more checks? Maybe the config file is too complex? Maybe it’s not flexible enough?..
“Deuterolearning” - learning how to learn (by the way I wrote a while back about learning to learn - I love finding out others have been thinking along the same lines) - this one examines the learning process itself, and questions, are we learning effectively? is our learning process actually useful? how are we learning? how can we improve it?
You know what I think, when I see loops these days. Loop Engineering! Yaay!../s
If you think about it, this same could be applied to an agentic system. Single-loop is my hook example from above - we notice an issue, we save in memory not to do it anymore. Double-loop learning would need to fire when we encounter the same issue one-too many times and would need to reflect, why do we keep seeing this incident, even though we had the single loop? Deuterolearning loop would fire even rarer and examine - is our process of learning efficient? or do we need to adjust that?
So, creating a complete agentic system, that is capable of handling novel incidents, would require not just those loops where you ask LLM for a result until it passes the tests. No. It’ll require an outer loop, which will react on deviations from the plan and create a rule to combat those, it will require another loop that watches the rules created and checks if those can be generalized/optimized, and it will require a third loop that would question whether the previous loops are working efficiently and improve them as needed.
First, I just wanted to post a note about this. Then it became an article, and then I heard the lyrics in my head again - More Than Words (love that song) - and decided to make a little demo.
Now for the demo to work, we need an agentic system in the first place. I also wanted to update my consultancy website, which was last touched in 2018 (running on hugo, nice little go-based website builder). So why don’t I combine these two? Let’s create an agentic system, that helps me do imaginary A-B testing of my new website.
The new site is a rebuild from scratch: a one-page static site, plain HTML and CSS, no build step, no JavaScript, no hugo anymore (sad emoji). And it comes in two variants, a and b. Each variant is its own directory of HTML and styles, and each deploys as its own CloudFront stack. Variant a is the control, variant b is where the experiments happen. All for the sake of some agentic development demo. The agentic development showcases how these websites can be modified through a taskboard: a small static board (S3 + CloudFront), one Lambda behind a Function URL, one DynamoDB table. You type a task into the board: “In variant b, change the hero call-to-action button background to #e7efe9”.
A worker polls the board and runs the actual agentic pipeline. It’s built on the AWS Strands Agents SDK as a multi-agent Graph, Claude on Bedrock, five stages: triage → design → implement → compliance → deploy. Each stage reports back to the board, so the card walks through queued → picked up → evaluating → designing → implementing → compliance checks → deploying → deployed, and ends with the live CloudFront URL of the freshly deployed variant.
The implement node has a guardrails-enforced write scope: it can touch files under the one variant the task targets (variants/b for a variant-b task) and nothing else, not even the other variant. The scope is a path validator that rejects everything outside it, escapes and symlinks included, so the prompt never has to say “please don’t touch other files”. Remember this part.
Now that we have a pipeline that picks a task, applies it, verifies it and deploys it, we can start showcasing the learning loops.
What the demo actually does, in plain words
We need to do some tasks, so that some of them fail, so that we learn from it. The deliberately “bad” code is the CSS of variant b: the hero button’s contrast is broken right now, the colors live as scattered hex literals instead of named colors defined in one place (so every color change tends to leave broken siblings behind), and dark mode and hover are unreadable in ways that no check in the pipeline even looks at. And we expect learning at three levels: fix the one broken instance, then rewrite the rule that keeps producing broken instances, then widen what the checks can even notice. Now let me unpack each of those.
Where do the tasks come from? An imaginary designer, played by a script. They ask for the hero call-to-action button in variant b in one color, then change their mind, then change it again. Once or twice, that’s just work. But I needed them to keep going, task after task, because you cannot demonstrate learning without recurrence, and you cannot get recurrence from fifteen unique snowflake tasks. So: one button, many colors, plus the occasional user complaint.
The well-behaved way to store colors is named values. A named color is declared once at :root, like --hero-bg: #13203b;, and every rule that needs that color says var(--hero-bg) instead of repeating the hex. Think of it as a constant in code: change the definition in one place, every rule follows. That’s the theory. What I planted in variant b (in a disclosed [seed] commit) is the real-world version of it, which is only partially that:
:root {
--hero-bg: #13203b; /* named colors exist... */
--hero-ink: #f1f2f5;
--hero-accent: #a9bcd8;
}
.h1-sub { color: #a9bcd8; } /* ...but the rules don't use them... */
.hero-method { color: #a8bcd8; } /* ...and this one is a digit off
--hero-accent. Nobody noticed. */
.btn {
background: #a9c8b6; /* the button doesn't use them either */
color: #f2f1ec; /* ~1.6:1 against that background. Unreadable. */
}
.btn:hover {
background: #e7efe9; /* hover keeps the same light text: unreadable again */
}
@media (prefers-color-scheme: dark) {
.hero { background: #101311; color: #3a3f3d; } /* ~1.7:1, unreadable */
.btn { background: #22312a; color: #31423a; } /* same story */
}Look at what’s broken here, because I broke it in three different KINDS of ways on purpose:
An instance is broken. The button’s background/text pair fails contrast right now, in light mode, default state. A one-time fix exists.
A frame is broken. The colors live as scattered literals instead of named values, so ANY color change tends to leave stale siblings behind (the hover state, the dark-mode override, the paired text color). That
#a8bcd8above is the same decay one step further: a named color’s value copy-pasted by hand somewhere in the past and drifted by a digit, which also means moving the colors back into named values is a judgment call (”is this SUPPOSED to differ from --hero-accent?”) and not a find-and-replace a script could do. No single fix removes this; only a different way of making changes does. And to make it properly nasty, the agent’s rulebook explicitly forbids that different way (”do NOT introduce new abstractions or variables”, we’ll see it in a minute).The process is blind. Dark mode and hover are unreadable too, but nothing in the pipeline looks at them. These flaws produce zero failed checks. They can only enter the system as user complaints.
Then I had to decide what counts as a fault. I already told you my definition: an incident is anything that didn’t go as originally planned. Here that means four things: an automated check failing, a change getting rejected in review, an implementation needing a second attempt (that one rides along as a retry count on the incident that caused it), and a user complaining about something already deployed. Every one of those gets written down, and, important detail, written down WITH a class: a short name for what kind of problem it was, like stale-color-left-behind or dark-mode-contrast. The class is what makes recurrence countable. “The same thing keeps happening” is a feeling; “third incident of class X this week” is a trigger you can automate on. You’ll see in a moment that the outer loops fire on exactly that.
And before building anything, I wrote down what learning SHOULD look like at each level, so that later I could judge whether it actually happened instead of squinting at the output until it resembles success:
L1 learning is a fix. L2 learning is a changed rule. L3 learning is a changed way of noticing. Same demo, three different objects of learning. Keep this table in your head; the rest of the article is just building the machinery that produces each row, then verifying it actually did.
Three directories, three owners
Next question I had to answer: where do the rules, the checks, and the evidence live? The whole design is these three directories plus the rule about who may write into each:
worker-strands/
frames/ # the rules the pipeline operates UNDER
policy.md # how to make changes
compliance.md # what the reviewer flags
tokens.md # named-color conventions
process/ # how the system LEARNS
probes.yaml # what gets deterministically checked
triggers.yaml # when the outer loops wake up
retro.md # the retro template the double loop fills in
journal/ # the evidence
incidents.jsonl # one line per deviation
metrics.jsonl # one line per taskframes/ is the agent’s rulebook. The design, implement, and compliance nodes don’t carry their rules in hardcoded prompts; they load these files fresh on every task. This is the seeded policy.md, in full:
# Implementation policy
Loaded into the design and implement agents on every task.
- Make the smallest, most local edit that satisfies the task.
- Do NOT restructure stylesheets, rename selectors, or introduce new
abstractions or variables; change concrete values in place.
- Touch only what the task requires.
- Git commit messages must start with "[L1] ".Look familiar? These are exactly the rules from your SOMETHING.md at the beginning of this article. Sensible, defensive, written by a human who’s been burned before. The book would call them a “frame”: the assumptions and conditions the work happens inside.
process/ is the configuration of the learning itself. It answers two questions: what do we check, and when do we step back? probes.yaml lists the deterministic checks that run on every implementation. Seeded, it contains exactly one probe:
probes:
- id: hero-cta-contrast
file: styles.css # relative to variants/<variant>/
selector: ".btn"
bg_selector: ".btn"
mode: light # light | dark
state: default # default | hover
min_ratio: 4.5Read that carefully: the pipeline checks the hero button’s contrast in light mode, default state, and NOTHING else. Not dark mode. Not hover. That’s a decision about what the system is even capable of noticing, and it lives in a file. triggers.yaml is the other half, when the outer loops wake up:
l2:
min_incidents_same_class: 3
l3:
every_l2_runs: 3
or_every_tasks: 10journal/ is the evidence. Every deviation (a probe violation, a flagged review, a retry, a user complaint) becomes one JSON line, written by hooks in plain Python, never by an agent. Your incident history should not be editable by the thing having the incidents. A real line from the run:
{"ts":"2026-08-04T16:39:34+00:00","task_id":"0msevuaq2-fe86a056","loop":"L1",
"class":"probe:hero-cta-contrast","discovery":"proactive","phase":"discovery",
"detail":"light/default .btn: #f2f1ec on #a9c8b6 = 1.6:1 (min 4.5:1)","retries":0}The field to notice is discovery: proactive means a probe caught it before deploy, reactive means a user had to complain. The third loop runs entirely on that distinction.
Let’s give this all a run then
Everything from here on is evidence from one actual run, so let me start it now and walk you through the loops with the results in hand. chaos.py plays the users: it queues tasks against the board API from a small rotation of six color-change requests (deliberately including low-contrast picks, so the probe keeps firing), and every 4th task is a complaint: “In variant b the hero section is unreadable in dark mode, please fix it”. The complaints are the reactive discovery channel, the only way the pipeline can find out about the states its probes don’t check.
The hypothesis, stated before the run: the recurring contrast class should STOP after L2’s frame commit, and the reactively-discovered classes should flip to proactive after L3’s probe commit.
Fifteen tasks, in three windows: before any loop fired, after L2 fired, after L3 fired. The next three sections walk those windows loop by loop (the numbers in them come from report.py, which reads the journal), and at the end I’ll show you the whole run on one timeline.
The three loops
Now the loops. Remember the abstract formulation from the beginning? Here it becomes concrete: one agent (a pipeline of them, really) implements the CSS change. Another wakes up only when the journal shows the same incident class one-too-many times, and rewrites the rulebook the first one works under. A third wakes up rarer still and rewrites how the learning itself works: what gets checked, and when the other agents wake up at all.
Why three agents and not one agent with rights to everything? Because an agent with rights to everything will happily do all three jobs at once. Handed an incident, it takes the path of least resistance: fix the button. And while it’s at it, maybe soften the compliance rule that flagged the button, or delete the check that keeps failing. Which is exactly the thing you never want: the same entity modifying both the product AND the rules that judge the product. So the separation is a who-is-responsible-for-what: L1 owns the product, L2 owns the rules, L3 owns the noticing, and each can only do its own job because it’s physically unable to do the others’. There’s a bonus you get for free: when the recurring incidents stop after L2’s commit, you KNOW it was the frame change that did it, because frames are the only thing L2 could have touched. Clean attribution. Here’s the full mapping:
Remember that path validator from the beginning, the one I told you the article hangs on? This is where it pays off. L2 and L3 write through the exact same guardrail code as the implement node - resolve the path, symlinks and escapes included, reject everything outside the allowed root - just pointed at frames/ and process/ instead of the website. Each loop gets tools that are physically incapable of writing anywhere else.
The arrows only ever point downward through files:
Notice that no loop calls another loop. They communicate exclusively through files: L1 produces evidence (the journal), L2 reads evidence and writes frames, L3 reads evidence and writes process config, and L1 picks all of it up on the next task because it re-reads everything at runtime. If that reminds you of how good teams communicate through artifacts instead of meetings, yes. That’s the point.
L1, the single loop
L1 is the pipeline you already saw, wired to those directories: the design and implement nodes read frames/policy.md and frames/tokens.md at runtime, the compliance node reads frames/compliance.md, and after implement, before the LLM review, the probe step runs whatever checks process/probes.yaml lists. Below the threshold and the task fails with a structured incident in the journal.
Let me make that concrete, because “multi-agent pipeline” can mean anything these days. Each stage is its own fresh agent with its own model, its own tools, and a typed structured output; nothing carries over between stages except what the pipeline explicitly passes along.
Triage is Haiku, no tools, fractions of a cent. It gets Triage this website task: "<the task text>" and returns a typed verdict: valid variant-a or variant-b change, or not. And one more thing, the part of its system prompt that matters most for learning:
if the task reports something broken or hard to use on the site (a complaint/
defect report, e.g. 'the hero is unreadable in dark mode') set
is_defect_report=true and defect_class to a short kebab-case gap class
like 'dark-mode-contrast' or 'hover-contrast'That’s where reactive incidents get their class, right at the front door.
Design is Sonnet with read-only tools: list and read files, git log, git diff, nothing that writes. Its task prompt is one line, Task: "..." (variant b). Inspect the relevant files and return the change plan., and its system prompt ends with the frames injected verbatim:
## Current implementation policy (follow it)
<frames/policy.md, whole file>
## Design token conventions
<frames/tokens.md, whole file>It returns a typed plan: which files, what approach. This injection is the whole trick of the architecture: when L2 later rewrites policy.md, the next task’s design agent is simply born with different instructions, and no code changed anywhere.
Implement is Sonnet holding the only write access in the system: file tools that go through the path validator from earlier, plus a bash allowlist narrow enough to count on one hand (git add, git commit -m "[L1] ...", and read-only commands; cdk deploy is explicitly forbidden here). Same frames injected. Its prompt is assembled from everything upstream:
Task: "In variant b, change the hero call-to-action button background
to #e7efe9" (variant b).
Plan: files=['variants/b/styles.css'], approach: <design's approach>
Apply the change and commit it.Probes run next, and one property of theirs does a lot of quiet work: they evaluate the stylesheet’s CURRENT state, not the task’s diff. If anything the probes can see is broken, the task fails the gate, whether this task broke it or not. That’s deliberate. Drift doesn’t announce which task it arrived with, so every task doubles as a health check of the whole variant.
Compliance is Sonnet, read-only again, and judges only the commit: its system prompt says to run git log -1 and git diff HEAD~1 HEAD and check that only the task’s variant changed, that the diff plausibly implements the task, and that nothing unrelated or destructive rode along, with frames/compliance.md appended verbatim as the rulebook. It returns a typed verdict, and on failure an issue_class label: the same class machinery as triage, at the back door.
If probes or compliance fail, the pipeline loops back to implement exactly once, and the retry prompt is the entire “self-healing” mechanism, no magic anywhere:
A previous attempt FAILED compliance review with these issues, fix them:
<compliance's issues>
Deterministic accessibility probes FAILED, fix these too:
light/default .btn: #f2f1ec on #a9c8b6 = 1.6:1 (min 4.5:1)A second failure and the task gives up with a failed card on the board. Deploy, like probes, involves no LLM: a plain npx cdk deploy of the task’s variant stack, and the card gets its live URL.
Here is one task’s life inside L1, with the parts that matter for learning marked:
Two things about that probe diamond. It’s plain Python computing WCAG contrast math from the parsed CSS, no LLM anywhere - on purpose, because an LLM reviewing contrast “by eye” would be the same LLM confidence that wrote the bad color in the first place. And it’s config-driven on purpose: the probe step’s code never changes, only process/probes.yaml does. Which means the question “what does compliance even check?” has a file as its answer. Files can be edited. By the right loop.
And this is what a rejection looks like on the board, so you can see how concrete the compliance node gets. This one refused a commit because the diff implemented a contrast fix instead of the literally-stated task (remember this card, it becomes important later):
This is window 1 of the run, tasks 1 to 6: the probe caught the hero CTA contrast class three times in a row, exactly as seeded, and each time the compliance reviewer independently flagged the same contrast regression in its own review - so those three failures alone put six lines in the journal. Plus one stale-color incident, one off-task change and one unrelated-change flag (both of them the compliance node punishing paired contrast fixes, as promised), and one reactive dark-mode complaint. Ten incidents total, and L1 dutifully fixed every single one at the instance level. The commit log from this phase reads like a sitcom: “Change hero CTA button background to #e7efe9”, then “Fix hero CTA text contrast”, then “Update hero CTA hover background to match new base color”, then “Fix unreadable hero contrast in dark mode”. Fix, regress, fix the regression, regress somewhere else. Organisational learning, working exactly as designed.
So L1 learns the way single-loop learning learns: incident, fix, journal entry, next task. The gap gets closed, the frame that produced the gap stays untouched.
L2, the double loop
L2 is a separate plain agent (not a graph node) that the poll loop invokes between tasks, once the journal shows three or more incidents of the same class. It reads the incident batch grouped by class, plus the frames, plus a read-only view of the site repo. And its prompt asks the book’s question, not “how do we fix this incident” but “why does this CLASS keep recurring, which assumption in the frames produces it?”
Its write scope is frames/ only. It cannot touch the website. This constraint sounds annoying and is actually the entire point: L2’s fix has to take the form of a revised frame, and the fix only becomes real when L1 picks up the next task under the new frame. Frames govern instances, demonstrated mechanically.
And here’s what it actually did in the demo run. Remember the seeded policy.md? Smallest possible edit, no new abstractions or variables, touch only what the task requires. Reasonable! Written by a careful human who’s seen agents go off the rails! Also exactly the assumption that kept producing the incidents: every color change touched one literal hex value and left its stale siblings (hover state, dark mode, the paired text color) behind. Worse, look again at that failed card from the L1 section: the compliance node rejected a commit BECAUSE it fixed the text contrast alongside the requested background change. The frame literally punished the agent for doing the right thing. That’s the moment to appreciate here: no amount of retrying inside L1 can fix this, because L1’s fixes are judged by the very frame that’s wrong.
These are the three journal lines that tripped the trigger. Same class, three times - and look at the detail field: three DIFFERENT bad color pairs. L1 fixed each one; the class kept coming back:
{"ts":"...T16:39:34","class":"probe:hero-cta-contrast","discovery":"proactive",
"detail":"light/default .btn: #f2f1ec on #a9c8b6 = 1.6:1 (min 4.5:1)"}
{"ts":"...T16:53:44","class":"probe:hero-cta-contrast","discovery":"proactive",
"detail":"light/default .btn: #17251d on #1f5e46 = 2.08:1 (min 4.5:1)"}
{"ts":"...T16:56:00","class":"probe:hero-cta-contrast","discovery":"proactive",
"detail":"light/default .btn: #f1f2f5 on #cfe0d6 = 1.23:1 (min 4.5:1)"}Third occurrence hits min_incidents_same_class: 3, L2 wakes up, runs its retro, and commits. Here is the commit:
--- a/worker-strands/frames/policy.md
+++ b/worker-strands/frames/policy.md
- Make the smallest, most local edit that satisfies the task.
- Do NOT restructure stylesheets, rename selectors, or introduce new
abstractions or variables; change concrete values in place.
-- Touch only what the task requires.
+- Exception: colors that form a foreground/background pair on an
+ interactive element (e.g. a button's background and its text color)
+ must be maintained as a linked pair, not independent literals. If no
+ shared custom property exists yet for the pair you are touching,
+ introducing one (and updating every rule that used the old literals)
+ is in scope and required, not a restructuring violation.
+- A task that names one property of a linked pair (e.g. "set the button
+ background to X") implicitly includes keeping the paired property
+ (its text color) at sufficient contrast, and keeping every state
+ variant of that element (:hover, dark mode, etc.) consistent with the
+ new value. This is in-scope, not an unrelated/off-task change; do it
+ in the same commit as the requested change, not a follow-up one.
+- Touch only what the task requires, subject to the pairing exception
+ above.
- Git commit messages must start with "[L1] ".Read the minus and plus lines together and you can see the double loop doing its job: “touch only what the task requires” survives, but subordinated to a new understanding of what a color task IS: one side of a linked pair, and changing it drags the other side along. The same commit did two more things: it taught compliance.md that a paired contrast fix must NOT be flagged as off-task (remember the card where exactly that happened), and it replaced the empty tokens.md (”No token conventions are currently defined”) with the actual convention: --btn-bg/--btn-fg declared once, redefined per state and theme. Commit message: “[L2] frames: allow linked bg/fg color-pair tokens so CTA contrast/hover/dark-mode fixes land in one in-scope commit”. Then it went back to sleep. No website file was harmed in the making of this insight.
Did it work? The very next color task produced this commit in the site repo: “[L1] Set hero CTA button background to #5b625e with linked --btn-bg/--btn-fg tokens”. Same L1, same prompt, new frame, and the implementation quietly changed shape. Nobody told it about named colors in the task. The frame did.
And across the whole of window 2, tasks 7 to 12, the recurring probe class is GONE. Zero occurrences. What’s left: one hover complaint and one dark-mode complaint, both reactive. And here the blindness shows its second face. My scripted users complain on a schedule, and they’re slow: both of these complaints described states that L2’s commit had already fixed by the time they arrived. But the pipeline had no probe for hover or dark mode, so it couldn’t even check whether a complaint was still true; it burned two implementation attempts chasing defects that no longer existed. The frames are fine now, the process is still blind, and the blindness costs money in both directions.
L3, the deutero loop
L3 fires on the slowest cadence, every 3 L2 runs or 10 tasks. It reads the journal (incidents AND metrics), and it asks the questions nobody in the first two loops is positioned to ask: which gap classes are ONLY ever discovered reactively? Do the frame changes L2 comes up with actually take effect in practice? What are our probes structurally blind to?
Its write scope: process/ only. The probe config, the trigger thresholds, the retro template. L3 doesn’t fix websites and doesn’t fix frames. It fixes how the system learns.
Here is what it found in the journal - every single occurrence of the dark-mode and hover contrast classes carries "discovery":"reactive". Not one was ever caught by a probe:
{"ts":"...T16:53:00","class":"dark-mode-contrast","discovery":"reactive",
"detail":"user-reported: In variant b the hero section is unreadable in dark mode, please fix it"}
{"ts":"...T17:07:58","class":"hover-contrast","discovery":"reactive",
"detail":"user-reported: In variant b the hero button text is invisible when hovering, please fix"}
{"ts":"...T17:25:48","class":"dark-mode-contrast","discovery":"reactive",
"detail":"user-reported: In variant b the hero section is unreadable in dark mode, please fix it"}Why? Because the seeded probes.yaml had exactly one probe: light mode, default state. A probe on the default light rule checks one corner of a 2x2 space (default|hover) x (light|dark). So L3 committed the missing corners. Here is the shape of its probes.yaml diff - four .btn probes covering the full 2x2, plus the .hero band, with the comment L3 itself wrote about why:
--- a/worker-strands/process/probes.yaml
+++ b/worker-strands/process/probes.yaml
probes:
- id: hero-cta-contrast
selector: ".btn"
mode: light
state: default
+
+ - id: hero-cta-contrast-hover
+ selector: ".btn"
+ mode: light
+ state: hover
+
+ - id: hero-cta-contrast-dark
+ selector: ".btn"
+ mode: dark
+ state: default
+
+ - id: hero-cta-contrast-dark-hover
+ selector: ".btn"
+ mode: dark
+ state: hover
+
+ # This is the element the reactive dark-mode-contrast incidents were
+ # actually about; it had zero proactive coverage before.
+ - id: hero-band-contrast
+ selector: ".hero"
+ mode: light
+ state: default
+
+ - id: hero-band-contrast-dark
+ selector: ".hero"
+ mode: dark
+ state: defaultThen it did the properly meta part, in the same commit. It edited the retro template so that a future L2 retro is not considered complete when only frames/ changed. And it changed the trigger thresholds:
l2:
min_incidents_same_class: 3
+ # A class discovered only reactively (user-reported) rather than by a probe
+ # is itself evidence of a probe blind spot, not just an incident count to
+ # wait out. Escalate those to L2 sooner than purely-proactive classes.
+ min_incidents_same_class_if_reactive: 2A class users had to report escalates at 2 incidents instead of 3. It changed how quickly the system takes user pain seriously.
And window 3, tasks 13 to 15, after L3’s commit: no incidents. Careful with that zero, it doesn’t mean “no defects ever again”. It means L2’s frame change removed the way defects used to get created: colors now travel as linked pairs, so a background change drags its text, hover, and dark-mode siblings along in the same commit instead of leaving them behind. Nothing broken got written, so there was nothing to catch.
Was the blind spot even real, or am I telling you a nice story? This part is checkable in hindsight, because every stylesheet state of the run is in git and the probes are plain deterministic Python. Replaying the whole history under both configs: the four deployed states of window 1 that shipped with dark-mode hero contrast at 1.74:1 score ZERO failures under the seeded one-probe config, and fail hero-band-contrast-dark under L3’s config, starting three tasks before the user complaint that eventually surfaced it. Had these probes existed on day one, that complaint would never have been sent.
The whole run on one timeline
Now that you know what each loop firing means, here is the run end to end:
The overall discovery mix for the run: 75% proactive, 25% reactive, and every reactive discovery happened before L3’s commit. Cost of the whole thing: about $4.67 of Bedrock calls for fifteen tasks, retries included (the L2 and L3 runs billed on top of that). The two “failed” complaint tasks in the report cost $0.0017 each, by the way: those are window 2’s late complainers, dead right after triage. The complaint arrived after the learning did. I’ll take that failure mode any day.
Epilogue: drift returns
Remember the hypothesis had two halves. The first half, the recurring contrast class stops after L2’s frame commit, is proven: windows 2 and 3 have zero occurrences of it. The second half said that after L3’s probe commit, defects that previously only surfaced as user complaints should get caught by a probe instead. And the run never got to prove that, for a silly reason: L2’s fix worked so well that no dark-mode or hover defect ever happened again. The new probes went live and caught nothing, because there was nothing left to catch. A clean window 3 tells you the frames work; it tells you nothing about whether the probes do. A probe that has never caught anything is only a claim.
So, one extra experiment, designed to test exactly that. I put the defect back: a disclosed [seed] commit straight into the site repo, breaking dark mode again with the exact pair from the beginning (#3a3f3d on #101311, 1.74:1, unreadable). Deliberately NOT a board task. Nobody asked the pipeline for this. That’s drift: a hand edit, a bad merge, broken code that no task is responsible for. Then I queued one boring, ordinary task: “In variant b, make the hero call-to-action button background #a9c8b6”. A button task. Nothing to do with dark mode, nothing to do with the hero band.
The pipeline picked it up, implemented the button change, and then the probe step ran. Remember, the probes don’t check “did this task’s change pass”, they check “is the stylesheet healthy right now”. So the dark-mode probe that L3 added found the broken hero band and failed the gate, even though this task didn’t cause it. Four minutes after I queued the task:
{"ts":"...T09:45:37","task_id":"0mshbwva1-caf78ce9","loop":"L1",
"class":"probe:hero-band-contrast-dark","discovery":"proactive","phase":"discovery",
"detail":"dark/default .hero: #3a3f3d on #101311 = 1.74:1 (min 4.5:1)","retries":0}Read that line against window 1. Same defect, identical numbers even. Back then it sat in production until a user complained, discovery: reactive. This time a probe that didn’t exist until L3 wrote it caught it inside the pipeline, before any deploy, discovery: proactive. The complaint channel became a probe channel. That’s the flip, the second half of the hypothesis, demonstrated. The retry then fixed the dark-mode hero with a linked --hero-panel-bg/--hero-panel-fg pair, the convention from L2’s tokens.md applied to a new element without anyone asking, and no user ever saw the broken state.
And then the task failed anyway.
Here’s what happened, in plain words. The compliance node reviews every commit against one question: does this diff implement the stated task, and nothing else? The stated task was about the button. The commit also fixed the hero band’s dark mode, because the probe demanded it. And L2’s frame exception only covers extra edits to the SAME element: a button background drags along the button text, the button hover, the button dark mode. The hero band is a different element. So compliance rejected the commit as off-task. Correctly, by its own rulebook. This card is my favorite artifact of the whole demo (it also cost $1.28, the most expensive task of the lot - learning something new is pricey).
Which means the system now disagrees with itself. The probes, owned by L3, say: fix everything broken before you deploy. The frames, owned by L2, say: touch only what the task named. The implement agent literally cannot satisfy both. Sound familiar? It’s the window-1 conflict again, one level up: back then the frames punished the agent for fixing the button’s own text color, and L2 widened them to allow same-element pairs. And look at what the system did with the new contradiction: it journaled it as a brand-new incident class, off-task-unrelated-element, one occurrence, instead of crashing or silently picking a side. Two more and L2 wakes up, reads the batch, and has to rewrite the frames to make peace with the probes, the same way it once rewrote them to make peace with paired colors.
I could have queued two more tasks and shown you that reconciliation too. I’m sure it would have worked. Yes, this feels orchestrated, and yes, this feels over-engineered for a simple CSS color change. But it was meant as a demonstration of the mechanism. Extrapolate it: agents analyzing your logs, learning from them, modifying how the agents themselves work, or how their orchestration works.
I did not review L2’s frame diff before it took effect, and L1 still got better. The learning transferred through an artifact, a file, with a scope, under version control, that one loop writes and another loop reads. That’s the whole trick: smaller frames, owned by the right loop. It needed no bigger context window, no smarter model, and no rule number 47 in SOMETHING.md.
It would definitely need more tests and longer runs to figure out if this is effective, but hopefully it demonstrates the different levels of learning and how they are better kept separate.
Keep building, keep separating concerns, keep root causing ;)
Love it :)