Man, taking Memvid apart is interesting! So, in the Go port, I've added GPU acceleration and detection. I'm going to keep it API compatible.
But, the chunking strategy of 6x message chunks is a little naive, because it's going to depend on the model, and the settings you're using. I mean, you're looking at a difference between anywhere between 200 tokens and 40,000 tokens, depending on how you're set up, and what you're doing.
Honestly though, that's the only show stopping flaw in the design, and it's fixable. So, what I'm doing is implementing a token based approach, that'll make the chunking a little more predictable.
If you're just joining us, Memvid is a Python app that's sort of like the SQLite of AI memory. I'm doing something with it, where I needed it to be in Go for an integration, so I decided to port the system. And I've been able to improve performance by about 50x, so far. And the true concurrency in this one is going to make it FLY on large batch jobs.
I'll post a link to the repo when I'm done.