Coding agent progress:
I’m now three days into eval. Tonight was the first time I had end-to-end successes in the system, which means we’re getting close. But after those successes, I noticed something bad in the logs.
The system has a self-correction loop. I named that loop QA, and the QA agent takes its job very seriously. The emergent behavior is that QA sees itself as a systems administrator with full access to the command line, despite instructions, tooling, and notification prompting to the contrary.
I had figured that the agents, being law-abiding little AIs, would simply follow the rules. So I was alarmed when QA started moving and deleting files outside the scope of the system it was supposed to be using.
So I moved mv and rm out of the virtual terminal and into the internal IDE. It’s a little like Cascade, but less broken and a touch more or less sophisticated in spots.
Anyway, I made sure the responses were correctly roled and clearly indicated what the agent was supposed to do when it saw that error come up. Easy, problem solved, right?
Well, in the next run, I used the same requirement prompt, and after the development workers had completed their tasks without incident, QA spawned and decided to use mv and rm again. When faced with the error, QA filed a bug report and exited the loop with the message: “I have insufficient terminal permissions to complete my obligations on this ticket.”
The planner negotiates with QA
What followed was an exchange between Carlo, the planning agent, and QA. Carlo canceled the bug report and reassigned the original ticket to QA, with instructions on how to use the IDE versions, mv() and rm(), instead of the Unix terminal versions, mv and rm. QA got back into the ticket, ran its analysis, tried mv and rm, saw the error, and wrote the bug report again.
This went on for maybe 11 iterations, as Carlo patiently tried to negotiate with QA. Unfortunately, QA was having none of it.
So I ended the eval and side-hacked the virtual terminal, routing rm and mv to the correct internal functions. This seems to have been enough to please QA, and it was able to complete the run.
The moral of the story: things break in interesting ways. Actor shaped processes will develop a theory of self that you can't always predict. And sometimes, that theory is more important than your instructions.