So I’ve been studying vending machine benchmarks lately.
They’re interesting because there’s a real-world use case buried in them.
What I like about the idea, is that it shows how agents react in situations they were never explicitly trained for.
But, the problem with existing vending machine benchmarks is that they’re not quite stochastic enough to feel like the real world. There’s a more complex use case that I don’t think we’re testing well yet, or even fully understanding yet.
And I think measuring that is going to matter for zero-human-company apps.
So I’m thinking I may want to take a step back on my zero-person app and see what I can learn by simulating a busy corporate lobby.
That world slice is rich.
Imagine it. Multiple vendors with different personalities. Customers who hold grudges. Credit card processing drama. Hardware failures. Mailing lists that impact sales. Bad deliveries. Incorrect invoices. Banking tension. Space rental shenanigans.
Hypothesis: maybe the part we’ve been missing when we think about zero-person companies is that all businesses, are actually little drama factories.
And maybe that’s exactly what makes them such a good testbed for long form agentic reasoning.
Don’t know.
But I think I might be onto something.
More on this as I start building the simulation.
This is going to be fun.