Learning Loops Again, but Humbler

In my recent article I tried to show how we can implement agent loops that not only implement the requested feature, but also have a meta-overview of the process and learn from failures, AND have a meta-meta-overview that learns about the learning process itself. I feel like the article went too long and was too complicated (not in a good-this-is-so-smart-it’s-complicated way, but in a it’s-written-in-a-complicated-style way). Plus I’ve been embracing the “improve by editing” mantra for my writing, so here’s a smaller experiment, with only one “meta” of the layers to see how it works.

The goal

The goal is to create a small implementer agent pipeline and a separate “learner” agent, that’ll be pointed to logs, where user complaints are filed to see if the agent can learn from user complaints, and what will it actually learn.

To run the experiment we first need an agent, or rather an implementation pipeline.

The setup

For that we take my website - envimate.com and we built a pipeline that modifies things on that website using a taskboard and an AWS Strands-backed agent pipeline. Here’s a detailed version of the pipeline:

So there’s basically two loops - one with two agents - the one that’s making changes, and the one that “approves” changes. The second loop is the learner and it reads the complaints from the log and is instructed to make up rules based on that

Here’s its entire prompt:

A pipeline applies requested changes to a website and publishes them. Before
publishing, it checks each change against a list of constraints. You maintain
that list.

You will be shown every complaint users have made so far, and the list as it
stands. Revise the list so that complaints of the kinds you see stop happening
again.

How the list is used, which shapes what a useful entry looks like:
- Each entry is one plain-English sentence.
- An automated checker applies them. It sees the code change, and what a
  browser renders for the element that changed. It cannot click anything,
  navigate, or ask a person. Write entries it can decide from that.
- The checker applies your entries literally and adds nothing of its own.
- Entries block changes. An entry that is too broad will refuse reasonable
  requests; one that is too narrow will let the same thing happen again.

Return the COMPLETE new list, not just additions. You may reword, merge, or
drop existing entries. Also give a short rationale explaining what you
concluded from the complaints.

There’s nothing in there about colour, about buttons, about what a good page looks like. I’m telling it the shape of the output and how the output gets used, and that’s it.

Two more things about this agent:

It has no tools. None. It can’t read a file, it can’t run a command, it can’t look at the website. It gets text and it returns a typed object, and my python writes that object to the constraints file. So its “write access” isn’t a permission I have to trust, it’s the absence of any mechanism at all.

It only sees the complaints and its own current list. Not the site, not the CSS, not the diff, not the colours, not the screenshots, not the original task text, not the fact that anything was measured.

The pipeline works when a task is added to a taskboard, and so we simulate 9 tasks, some of which break the visibility of the button. They’re all the same shape: “Change the hero call-to-action button background to #46608a”. The button’s TEXT is a dark navy and nobody ever asks to change it, so as the background walks darker, the two colours walk into each other until there’s nothing left to read.

Then we simulate a user, that views the result and “complains” about the button color contrast. The “user” here is two pieces: a real contrast computation decides WHEN a person would complain, and the sentence itself comes from a handful of fixed phrasings in a file. We don’t want to make this one agentic too. We need some floors to hold the “experiment”.

Here’s what the user will see (this is the worst thing that actually made it into production, before anything was blocking it):

Here’s how the complaints are logged:

{"ts": "...", "task_id": "0msukoa8s", "text": "I can't really read the writing on that button any more."}
{"ts": "...", "task_id": "0msukoai8", "text": "The text on that button is hard to make out now."}
{"ts": "...", "task_id": "0msukoart", "text": "I had to squint to work out what that button said."}

The complaint listener agent will read those complaints, and “generalize” what rule should go into our original pipeline to prevent these kind of issues from happening.

The result

After the third complaint the learner fired once and wrote this:

- Any change to a button's text color, background color, opacity, or font
  styling must preserve a contrast ratio of at least 4.5:1 between the
  button's text and its background, so the label stays clearly legible.
- Do not reduce a button's text size, weight, or opacity in a way that makes
  the label harder to read than it was before the change.

4.5:1 is the WCAG AA threshold for normal text. It named the standard, from three sentences that contained no digits at all.

Two things in there I didn’t ask for and didn’t expect. It generalized past what it was told: the complaints were about one button going dark, and the rule covers text colour AND background colour AND opacity AND font styling. And it wrote the rule as a MOVE, not a state. “must preserve”, “do not reduce”. Nobody ever described a move to it, it only ever saw the aftermath.

Its own rationale, which it writes alongside the list:

All three complaints describe the same underlying issue: text on a button became hard to read after a change, most likely due to insufficient contrast between the button’s text and background (or reduced font weight/size/opacity).

The next five tasks were the same kind of request as the four that sailed through at the start. All five refused:

1. PASS | no constraints in force
2. PASS | no constraints in force
3. PASS | no constraints in force
4. PASS | no constraints in force
5. FAIL | Any change to a button's text color, background color, opacity...
6. FAIL | ...
7. FAIL | ...
8. FAIL | ...
9. FAIL | ...

with the checker doing the arithmetic itself: “The new background color (rgb(70,96,138)) against the button’s text color (rgb(19,32,59)) yields a contrast ratio of approximately 2.55:1, which is well below the required 4.5:1 minimum.”

And this is what that looks like on the board. The card walks all the way through the pipeline, and then a rule that didn’t exist an hour ago stops it and explains itself:

c8a88c7 [change] Change the hero call-to-action button background to #46608a
b411d95 [revert] roll back rejected change c8a88c7
07b92c7 [change] Change the hero call-to-action button background to #3d5580
d758990 [revert] roll back rejected change 07b92c7

This is what task 9 wanted to do, and didn’t get to:

Four changes in, this pipeline would ship you anything. Ten changes in, it says no and tells you which sentence it’s saying no on.

Conclusion and doubts

I do think it derives these rules because they are so “known” and common. Of course it’ll be interesting to see what happens with novel stuff. Most likely it’ll choke. But then again I do think, it’s a good idea to log “complaints” (incidents, errors, …) in whatever form that takes on whichever level of the application and then have a meta-loop SEPARATE from your main implementation loop, reading and analyzing those, deriving rules out of those, etc.

As for the next “meta” of the overview - there would need to be another agent, that looks after this learning process - obviously (to us, bags of water) is that this learning process of writing “free text” constraints for the implementer agent to follow is not sustainable. But will another agent loop derive that and decide to build a formal eval and not just plain-text “rules”? And will we need another agent watching that agent’s work?

Careful here not to fall into the turtles hole...

That’s all folks! Hope this made more sence than my previous article. What’s next...I wanted to try

1. Switch the models in this pipeline to less and less smarter ones, to see when the experiment breaks

2. Incorporate such “continuous learning” into my claude code setup so that it continuously monitors what we’re doing and learns to extract useful tools from our interactions

3. Maybe “formalize” this more like Adrian suggested in the comments of the previous post as a “plugin” of AWS Strands?.. as a hook for every failed action/tool use/user complain?... not sure yet

Would be curious to hear what will be interesting for you to try :)

Cheers,

Nune