Skip to content

Designing a deliberately vague agent evaluation

So I've decided that next month, unless something goes horribly wrong, I'm going to give the google ultra gemini service a spin. Just signed up for it now. Actually a lot more impressed than I thought I would be with Jules. I like the way it sits at the back of my process, rather than the front.

I feel like, generally, the kind of tooling I want is a world where I can give detailed explicit instructions to a machine, have it spin up as many threads as it needs to do whatever thing, and focus on whatever else I need to focus on.

I do not like human-in-the-loop processes that need me to micro-manage and constantly validate the machine every five minutes. Just build it, and let me review and test the whole thing.

Tonight's test involves high level directions.
I'm going to tell it to
1 - build an app.
2 - do a code quality and security audit on the app.
3 - write unit tests, thoughtful code comments detailing why decisions were made
4 - a small app upgrade, but written in vague high level, possibly confusing language.

And now we play!