Small update from the Shmoosh mines:
I'm testing packed attention-cache policies for local image generation on a 4070.
The first obvious question was: does it make things faster?
The honest answer is: not dramatically yet.
On the 1024 suite, the current best exploratory policy gets about a 1.048x mean speedup. That's real, but it is not a show stopper by itself.
The policy surface
The more interesting part is what happened structurally.
I tested different self-attention gates, and the 70% self gate was the first composition policy that felt correct rather than just numerically acceptable.
The 50% self gate:
min PSNR: 47.49 dB
mean speedup: 1.015x
The 70% self gate:
min PSNR: 49.17 dB
mean speedup: 1.048x
On the reading-nook case specifically, PSNR recovered from 48.82 dB to 50.08 dB.
That matters because it suggests the policy surface is real.
Cross-attention can activate earlier.
Self-attention needs to wake late.
That's important. I'm not at magic image-generating speeds yet, but the system is controllable and behaving the way I expect it to.
The current read is:
Packed encoding works.
Vectorized bit packing removed a chunky bottleneck.
Cross-attention looks like the likely production win.
Self-attention is quality-sensitive, and right now it is more of a late-stage fidelity intervention than a true speed policy.
What gets tested next
The next slice is kernel and policy-cost work: profile the self-attention streaming path directly, then test QJL64 and no-QJL K6/K7 variants for the 1024-token self-attention modules.
That should tell us whether self-attention can become a real speed win, or whether it should stay as a careful late-stage quality tool.
We'll get there.