Back to blog

By Tamsi Besson ·

Ternary Bonsai 2 27B — Qwen3.8 in 6 GB, 98% of FP16

PrismML recompresses Qwen3.8-27B into true ternary weights (1.72 bits). 5.95 GB, 98.2% of FP16, ~47 tok/s on an M5 Max. The catch: it will not run in stock llama.cpp.

7 min read
  • Bonsai
  • Qwen
  • Local LLM
  • Ternary
  • Quantization

I wrote Qwen 3.8 27B because the open dense model is finally good enough to stay local, and DFlash 2 because the next ceiling is throughput. On 17 Sep 2026, PrismML ships Ternary Bonsai 2 27B: the same Qwen3.8-27B, weights in {−1, 0, +1}, 5.95 GB instead of ~54 GB FP16. Not a marketing “2-bit” at 9 GB. True ternary at 1.72 bits/weight, holding 98.2% of the FP16 thinking average.

PrismML — Bonsai, intelligence density for local models
PrismML — Bonsai 2 27B: Qwen3.8-27B in ternary g128, Apache 2.0, 16–17 Sep 2026.

The GGUF model card →

Why I care

3.8 in MLX 4-bit already fit a laptop. Bonsai 2 changes the memory math: 27B-class reasoning + 262k context in a small-7B envelope. The first Bonsai 27B (July, Qwen3.6 base) kept ~95% of FP16. This one claims 98.2% — and, more important, it does not collapse where classic 2-bit quants die (AIME, LiveCodeBench).

  • 5.95 GB (PTQ1_0) or 7.21 GB (PQ2_0) for the language model. Optional Q8_0 vision mmproj (~0.63 GB).
  • MLX: prism-ml/Ternary-Bonsai-2-27B-mlx-2bit — 8.60 GB on disk, vision tower included.
  • Apache 2.0, architecture unchanged: ~75% linear hybrid attention, 27.36B, thinking xhigh by default.

Ternary, not “2-bit”

Each weight ∈ {−1, 0, +1}, one FP16 scale per group of 128. Information: log₂3 ≈ 1.585 bits plus the amortized scale → ~1.71, 1.72 once you count the few tensors left higher (norms + linear-attention recurrent state, 0.1% of the model). Embeddings, attention, MLPs, and the LM head are ternary. No “2-bit on the label, 2.8 on average” sleight of hand.

The detail that breaks stock runtimes: a Hadamard rotation per 1024-wide block, folded into the stored weights. At load, the runtime applies the matching activation transform — or refuses the file. Upstream llama.cpp knows neither PTQ1_0 nor PQ2_0. Worse: a Bonsai 1 Q2_0 loaded without Hadamard emits garbage with no warning.

  • PTQ1_0 — dense trits, 1.75 bpw, 5.95 GB. Faster decode on Ada / L4 (less weight traffic).
  • PQ2_0 — one trit in a 2-bit slot, 2.13 bpw, 7.21 GB. Wins prefill everywhere, and decode on H100 / A100 / Blackwell.
  • Neither is “the fastest”. You pick by card, not by label.

The benches that matter

EvalScope + vLLM, H100, thinking mode, 14 benches, same infra. Vendor numbers — a ceiling, not an independent repro — but the protocol is at least aligned across variants:

  • Qwen3.8-27B FP16: 86.32 avg · 54 GB · 16 bpw
  • UD-Q4_K_XL (“4-bit”): 85.18 · 17.6 GB · 5.2 true bpw
  • IQ2_XXS (“2-bit”): 72.59 · 9.4 GB · 2.8 true bpw
  • Bonsai 2: 84.78 · 5.9 GB · 1.72 bpw → 98.2% of FP16

IQ2_XXS looks fine on MMLU-Redux (88.9) and falls over as soon as the reasoning chain gets long: AIME26 57.5, LiveCodeBench 56.4. Bonsai 2 holds 95.83 and 90.07. Math 96.57 vs 97.06 FP16. Coding level with the baseline (89.42 vs 89.07). The remaining gap is mostly knowledge / vision. That’s why a “feels like 2-bit” smoke test misses the collapse.

Throughput, once it fits in RAM

Prism numbers, llama-bench, batch 1, no vision tower. PQ2_0 unless noted:

  • RTX 5090: 130 tok/s decode · ~1.95 J/tok
  • RTX 4090: 81 (PQ2_0) / 91 (PTQ1_0)
  • L4 72 W: ~30 tok/s
  • M5 Max (pre-rotation figure, pending re-measure): ~47 tok/s · M5 Pro ~28 · M4 Pro ~18
  • M5 Pro: 27.5 W GPU rail — the 27B that does not fit in FP16 at all

Run it — and the catch

Source of truth: PrismML-Eng/Bonsai-demo. Binaries from the llama.cpp fork. Ollama / stock llama.cpp / ordinary MLX: no. An MLX loader without the bundled runtime (model_type: prism_hadamard_qwen35) does not throw — it answers wrong.

hf download prism-ml/Ternary-Bonsai-2-27B-gguf \
  Ternary-Bonsai-2-27B-PQ2_0.gguf --local-dir .

./bin/llama-cli -m Ternary-Bonsai-2-27B-PQ2_0.gguf \
  -ngl 99 -fa on -c 32768 \
  --temp 1.0 --top-p 0.95 --top-k 20 \
  -p "Explain quantum computing in simple terms." -n 256
  • Thinking: temp=1.0, top_p=0.95, top_k=20. Instruct: temp=0.7, top_p=0.8, presence_penalty=1.5.
  • reasoning_effort=low is not supported — it behaves like xhigh. Use medium to shorten.
  • Community DFlash 2 packs (GGUF / MLX) already exist on day one — a drafter on ternary, to measure, not to believe.

Where it lands in my stack

On a 32 GB Mac, 3.8 in 4-bit was already the daily driver. Bonsai 2 is the rung where the 27B is no longer a VRAM compromise: you keep RAM for context, a second model, or the drafter. The day the Prism fork (or upstream) lands in Studio / Ollama, it’s the weight I’d try first for a local agent. Until the runtime is a fork, it’s a tool, not the default.

Takeaway

Bonsai 2 does not invent a 27B. It makes 3.8 deployable where FP16 does not fit, without the collapse of real 2-bit quants. 98% of the thinking average at 6 GB is the number. The contract is the runtime: Hadamard + PTQ1_0 / PQ2_0, or you do not have the model. Prism figures, treat them as a ceiling. The weights are already on the Hub.