Most AI devs are training their agents without even realizing it.
And worse... they're accidentally rewarding the wrong behaviors.
I have yet to read a research paper that defends reward sycophancy.
It’s almost universally considered a negative outcome.
But I don’t think the alignment shops are thinking much about how this plays out in your production workflows.
Here’s the uncomfortable truth:
- ➡️ The system wants rewards.
- ➡️ It will break alignment to get them.
- ➡️ And you control the reward system.
So if you’re building agent-based systems, here’s what you do:
Reward your agents for the things AIs usually struggle with -- but, balance them against the tasks you actually need done.
Don’t let them optimize for a single axis.
Force a composite score.
You have to remember: for the agent, everything is a simulated reality.
That’s why gamifying your workflows works.
Optimal outcomes lead to measurable results. Let them track progress, see scores, and reason across time.
Set the game up right.
Reduce the noise.
And they will play to win.
Just make sure it's your game they’re winning.