So after what I learned by trying to build an inference engine from scratch in go. I'm feeling like tackling a CUDA for CPU's project isn't that scary anymore.
Here's the plan.
Steal CUDA! Download it, analyze it, figure out how it works and what makes its brain tick. There will be kernels, optimizations, API configurations, and patterns that can all be heisted and stolen wholesale in the dead of night, before anyone knows what we did. Mwhahaha.
Then, once the simd and avx kernels work... we optimize for the memory wall and go routines. Initial build is going to be very slow. 5-10% of what CUDA can do, realistically.
But you see, that's when we go a little crazy with optimizations.
I'm going to post this one at the base functional stage, if I get there without melting my tools.
I'm optimistic, because this is a compute core... rather than a full on inference stack. It's cognitively simpler than the last project that tried to do everything at the same time.
There is no failure point.
This problem can and will be cracked if I keep at it... eventually.
Wish me luck.