So the battle with configuration Loras is over. I won that one. This kind of configuration for tool settings is the future of agentic systems, and I'm excited about it.
Total training time for 93 endpoints, plus 7 mcp's, a claude style tooling harness, and a full verbose system prompt is six minutes on the Spark, and it works beautifully. There's no ceiling. The agent will memorize tool instructions and free up 9k tokens for my 200k context window, no problem.
Next up, the inference layer. Text is no problem. At 4fp, it screams at a little over 300 tokens/s. But: getting the quantized version of the Gemma-27b model with multi-modal intact and a 16 bit image reading stack is................... not easy to set up. "Not easy" isn't the right way to put it. Needlessly complex, absurdly difficult. Yeah, that's better.
Half of me wants to give up on it, and call it good enough. But without multi modal, I can't make the agent generate and iterate on images. Generation is no problem, but iterating? The agent can't iterate if it can't see the outputs. I mean, come on. That's the one cool trick I wanted this system to do.
The idea is that I want an agent that can take image inputs, pass them to a multi-reference world model, and create storyboards for animated sequences and ad campaigns.
I'm very close.