If you were casting this as a boxing match, the outcome should have been obvious.
In one corner: BitNet-100B, the behemoth of edge AI. A 100B-parameter model running locally on CPU, backed by genuinely interesting ideas around native low-bit inference.
In the other: Qwen3.5-9B q4, the smaller, scrappier contender. Leaner, lighter, quantized to 4-bit, and on paper, not supposed to be the one throwing the heavier punches.
If you reason the way a lot of the industry currently does, your instincts would probably tell you the bigger model wins this easily.
That was not what I saw.
I ran a two-round coding eval comparing BitNet-100B against Qwen3.5-9B q4 on a simple Python utility: recursively scan a directory, hash files in chunks, group duplicates by SHA-256, ignore symlinks, skip unreadable files, and print only duplicate groups.
Round 1 was the initial implementation.
Round 2 was a revision pass using cordial QA feedback.
Part of what makes this interesting is that BitNet is not just “a big model on CPU.” Microsoft describes BitNet b1.58 2B4T as a native 1-bit LLM trained on 4T tokens, and its official inference repo highlights running a 100B BitNet b1.58 model on a single CPU. There is real architectural innovation here, not just brute-force compression.
The failure modes
What stood out was the failure mode.
BitNet-100B was polite and cooperative. It took feedback well, adopted suggested structure, and looked responsive at a glance. But the code remained structurally unsound: wrong data shapes, broken iteration logic, incomplete grouping, and code that still didn’t satisfy the spec after revision. It absorbed the vocabulary of the task better than the mechanics.
Qwen3.5-9B q4 behaved very differently. It was much more focused on the job itself. Its first pass was already substantially more coherent, and its revision looked like what you’d expect from a decent junior engineer: preserve the working structure, fix the weak points, and keep moving.
The runtime difference
One caveat: this was not a perfectly apples-to-apples runtime setup. Qwen was running in a custom llama.cpp GPU harness using the neural-llama package Sam and I built, while BitNet was running in BitNet’s own llama.cpp fork on CPU. Different runtimes, different suppliers, different optimization surfaces.
That said, practical outcomes still matter. In the environment we tested, Qwen produced substantially better code and revised it more effectively, while BitNet remained structurally weak.
The larger point is that coding performance doesn’t come from parameter count alone. It emerges from the whole stack: architecture, training mix, post-training, quantization choices, and the quality of the runtime and harness actually serving the model.
BitNet-100B is still fascinating as a systems milestone. But in this coding eval, Qwen3.5-9B q4 was the better programmer by a wide margin.