Some lessons learned from this weekend.
- AI inference, at the mechanical level is less like application design than it is the most mathematically precise form of device middleware you'll ever run into.
- Tensors and data shaping are fun! And I want to do more of it.
- Anyone who tells you we've hit the limits of what compute can do is lying to you. Current methods of AI inference rely on systems for doing things that even model makers don't fully understand the details of. It's a fact.
What I've uncovered here is that there are serious technical bottlenecks, artifacts, conflicting optimizations, and two decades worth of garbage in most AI inference systems that can and should be cleaned out with better, purpose driven architecture.
- If you have a background in devices and math, you can ABSOLUTELY vibe code one of these things, and shake the very foundations current day compute is built on.
- Don't expect to do it in five days.
- There are computational limits. We're not even close to them.