Should we review our agents?

I have been running multiple agents working on different things in parallel for a while now and it works, but they get so much work done that I have no chance of keeping up. Add to that that coworkers are doing the same thing and I am faced with a veritable wall of information that I need to go through.

Some have reasoned that the solution to this is to elevate ourselves over the code, don’t read it. Focus on the big picture. But how do I do that without knowing what is produced? Should I not critically review what agents produce?

The problem is that agents can produce code faster than we can review it and currently it seems like our understanding of what was built is being lost in the process.

Then I read this fantastic post by Valentina Servile, Should we still design code for humans?. I really connected with the idea that AI is not the same technological jump that compilers were, compilers removed a lot of the grunt work in development like AI but the difference is in a deterministic outcome. What she also voices is that we are replacing programming languages with something inherently ambiguous. Something I fear will also reduce our understanding of how the things we build actually work.

I accidentally started a lengthy discussion during a meeting at work on this topic when a coworker was showcasing the way ze had been working with agent orchestration. My question was along the lines of “How much do you need to supervise the agent and review the output?” which might seem innocuous but probably came across as being a bit Luddite.

I got some pushback on that, that by crafting good prompts and/or agent files we could improve the quality of the work the agents produce to a level where the code can be ephemeral to us. But do we still understand the code that is produced? Does it follow good design principles?

As Valentina pointed out we are trading structured and deterministic language, code, for unstructured and ambiguous natural language.

The problem with this has been known for years, take the old programmer/logician joke.

A programmer is asked to buy a loaf of bread and if they have eggs buy a dozen.
The programmer comes home with a dozen loaves of bread.

Today’s AI doesn’t make that mistake but it is far from this obvious when natural language is used. There are so many possibilities for ambiguity when describing complex systems with natural language. The problem might not be if the AI understands us but did we specify what we actually meant and do we understand what the AI decided based on what it understood?

For me the solution has been having agents review each others’ work, again slowing down the process but increasing quality. This helps me in that once I review it the formatting, tests, linting etc. has mostly already been fixed and to an increasing degree with heavier models also some of the more ‘abstract’ parts like edge-cases and interactions. Some will ask why humans should review if the agents have already reviewed each others work but the problem is still about AI making decisions, the reviewing agent can be just as wrong as the agent it is reviewing, but this strategy helps me get to the decisions that matter. It is the same way that I would use a linter, to remove the parts that I don’t care that much about to begin with.

This still means that I review the work the agent does, this is quite similar to how I have been working for years already, in the good old days when I still wrote the code myself I always reviewed any pull request I created as if I was reviewing someone else’s code. The only difference was that instead of adding comments I simply fixed what I found. Now I do the same thing with the agent, I write up what I find and then asks it to fix it.

Once the process slows down I can start wrapping my head around it again, not reviewing every line of code but getting the big picture. The agent reviewing each others’ work helps me get the time to understand what is happening. My workflow is slower but my understanding is better and I feel confident with it.