So there we are.
Finally got GPU's the multiple levels probing and self optimization working.
We spent all day getting the device orchestration, and the dynamic kernels, and the p4 quirks, and the undocumented headers, and all of the layers and bits and pieces of everything you have to do to work with GPU's up and running on a clean room implementation running and....
Fire up the GPU and it fucking works, you guys! It fucking works! Base implementation hits 13 gflops with no optimization, only basic memory management specific to things we did with the GPU and kernels.
Cool.
Not terrible for the first attempt, but a respectable showing, all things considered, after what it took to get here with no CUDA or ROCm.
The CPU regression
Then... Claude says, "This is a huge improvement over the 2 gflops we're getting from CPU" and my heart sank.
Fuck me with a thorny baseball bat.
Shit fuck shit!
It happened again.
I screamed at my screen, wanted to yell at Claude (but didn't, Claude has no intent of its own, and is programmed to do its best with the available information, and patterns).
I wanted to puke.
Previous implementation, the last one we benchmarked, before we added Metal and Neural Engine support hit between 43-250 GFLOPS on CPU, depending on the operation, the intensity of the math, and the type of workload.
As high as 90% of theoretical. We've actually done something like this twice now, with the same design.
Dig into the codebase to find we had hit the same project bottleneck we've been hitting.
The motherfucking stupid library pattern that Claude doesn't know how to manage properly, because Anthropic never trained Claude on complex libraries.
Claude literally replaced a novel state of the art memory management implementation with... seven nested loops.
Rebuilding the CPU layer
Best guess is that this happened last Saturday when we added Neural Engine support for Fortran. I should have checked the changes a little more closely. Been more careful about my repo than I was.
Nobody's fault but mine.
It's okay, you can't write something like a linear algebra library without intense emotional ups and downs.
This is how you know you're doing serious engineering.
We'll fix it tomorrow, when we reimplement the whole CPU layer again.
Switching to the Reference pattern.
It should go a little more smoothly from here.
If this were easy, someone would have done it by now.
I'm feeling hopeful.