I'm still blown away by the fact that Midjourney is on the cutting edge of image and video generation systems, but they're still using a prompt moderation design that was built out in 2022. In AI time, that’s like looking back to 1990.
Today, you shouldn't need to monitor image generation prompts at all anymore.
Why? Good question. Glad you asked.
Let’s start with why you’d think you want to moderate image generation prompts in the first place. Lots of good reasons. Moderation helps you stay predictably within the limits of the law, giving you that sweet, sweet Section 302 compliance—and the comfort of knowing you’re playing by the rules and not complicit in abuse.
That’s a net positive.
But the law is never enough. So the next thing to think about is liability. Platforms like app stores and credit card processors all have their own rules above and beyond the minimum legal requirements.
And if you want to get paid, you play with them.
Now… why would I have an issue with any of this?
What prompt moderation misses
Let’s get technical.
We had a couple of fun problems with prompt moderation from the beginning.
First, dumb fuzzy distance matching is either not enough—or way too much.
Simple enough, right? Just deploy an LLM process and call it good?
WRONG.
If you're testing these things properly, one thing becomes clear fast: deploying an LLM process to moderate raw prompts opens up all the usual security holes you already have when you expose LLMs. And worse—half the AI artists in the world are experts at exactly this kind of injection.
Sneaky stuff, too. I have this one friend who's amazing at it. The first time I let him test one of my systems, I about cried. It was either the first or second Space Judges implementation—I don’t remember.
This was before we started doing things like time scaling and input context for pre-deployment training, like we do now. Back then, these models were wide open, and we had to rely on prompt engineering alone to solve the problem.
I did finally outsmart him, but it took me two weeks and hundreds of lines of code.
Image generators illustrate so much about AI in one place. Like how sampling works in diffusion models. You go through a set number of sampling steps, normally, because the system has trouble giving you a coherent output on the first try. That’s going to vary between models—but it’s a revealing pattern.
We should be doing that with everything. Integrated correction and refinement loops that make everything sparkle.
The good prompt systems work with juries—a group of processes or review steps with different heat and creativity settings, and a synthesis step for a final decision. That’ll get you close to determining whether a prompt is actually offensive.
But in strict technical terms… will an offensive prompt always generate an offensive image?
No. Absolutely not. You can’t depend on that. Sometimes the scariest prompt in the world will generate an unremarkable watercolor of a duck, and nobody knows why. Too many steps? Wrong seed? The way the machine understands a particular misspelling or stack of words?
The models behave wildly. We just hope they behave wildly… with clarity. It’s a stochastic process. And in my opinion, prompt moderation is mostly just theater to make your credit card processor or app store feel better.
Will that be enough for a court? I don’t know. I’m not a legal expert. I think the argument for intent is extremely murky. But what do I know?
Even a solid prompt moderation layer won’t defend you against experts. And the actual criminals? They’re just going to go where they can get away with it. You probably won’t meet many of them.
When false positives become the product
Then there’s the psychology of startup founders. These guys, man. You tell them that moderation is important, and that the credit card companies are going to eat their lunch if it’s not handled? Oh—and that PCI compliance will come with its own brand of friendly hacking?
They turn it up. All the way.
They won’t tolerate any false negatives. But false positives? No problem.
New problem: you’ve just scared away all your shiny new users. Congratulations.
Because if you crank the dial too far in the name of “safety,” you’re not just filtering edge cases—you’re rewriting the entire creative landscape. You’re telling users: We don’t trust you with ambiguity. The system starts failing more often. And—ironically—that’s the kind of thing that makes PCI auditors more paranoid, not less.
Meanwhile, too many failures and your users run for the hills. Because let’s be honest: image generation is basically a commodity service at this point.
And all of this before we even talk about what a real image moderation system looks like... or why you want one.
So what does a real image moderation system look like in 2025?
It looks at the image.
Not the prompt. Not the metadata. Not a bag-of-words from a model that was already out of date last spring. You need a vision-language model—a VLM that can actually see the output and reason about what it might mean.
Sure, your metadata is useful. Store it. Index it. Feed it into a persistent store. But don’t confuse it for the thing you're trying to moderate.
The loop should work like this:
Prompt is sent to the image generator.
Image is generated.
VLM evaluates output for security and image quality.
If the image fails moderation, correct the prompt and regenerate. If it passes, return it to the user.
Break loop after a fixed number of attempts (three is usually fine).
And the best part? The human on the other side of this—the one paying to use your product—almost never sees any of it.
You’ve made your moderation system effectively invisible. You’re no longer worrying about alienating your customers because of some overzealous filter.
Doing it this way also sidesteps prompt injection entirely. The VLM doesn’t need to see the prompt, just the image. Smaller security footprint. Lower token budget. Fewer headaches.
If money’s no object, Claude Opus-4 is great for this. But it's expensive. Thankfully, there are tons of open-source options that will get you close enough.
Check the Hugging Face VLM blog roundup: https://huggingface.co/blog/vlms-2025?utm_source=chatgpt.com
Or… you could be like Midjourney.
"Our AI moderator is unsure about this image."
That’s not safety. It's plausible deniability at scale.
Good luck with that.