There’s a particular pleasure in an AI endpoint that hums away on a shelf instead of invoicing you by the token. For the past two months I’ve been running GLM-5.3-Flash — Z.AI’s 320-billion-parameter mixture-of-experts model — locally on two MSI EdgeXperts, the lunchbox-sized GB10 machines that share their innards with NVIDIA’s DGX Spark. It has genuinely replaced a cloud endpoint: the full 1,048,576-token context window, four concurrent requests, images and video in, tool calling, structured outputs, and an OpenAI-compatible API on port 8888. Z.AI’s own model card describes GLM-5.3-Flash as “approaching Claude Opus 4.8 on coding and agentic benchmarks” at a tenth of the price; my copy of it sits on a shelf next to the printer, draws desk-lamp levels of power, and has never once asked me to accept new terms of service.
But the hardware is only half of this story, and frankly the less interesting half. What fascinates me is how the software caught up — a seven-week group expedition across two giant threads on the NVIDIA developer forums — a thousand posts of excitement, disappointment, obsession and repair — that took the model from “technically fits, nobody will use it” to one of the best local inference setups I have ever used. It has a complete dramatic arc: excitement, a hangover, a resurgence, a revelation, and a resolution. Here it is, chronologically, with names attached, and with the engineering explained — because the engineering is the story.

The machine, and why two of them
Start with the silicon. The GB10 Grace Blackwell Superchip is a single package containing a 20-core Arm CPU — ten Cortex-X925 performance cores plus ten Cortex-A725 efficiency cores — fused to a Blackwell-architecture GPU (6,144 CUDA cores, fifth-generation Tensor Cores) over NVLink-C2C, NVIDIA’s chip-to-chip coherent interconnect. The point of NVLink-C2C is that CPU and GPU stop being two computers with a bus between them and become one pool of memory: 128 GB of LPDDR5x on a 256-bit interface, moving 273 GB/s, coherently shared. Add a ConnectX-7 network port rated at 200 gigabits, a 4TB self-encrypting Gen5 NVMe drive, and one petaFLOP of sparse FP4 AI compute, and you have the EdgeXpert MS-C931: 151×151×52 mm, 1.2 kg, a honeycomb intake grille for a personality. It is not a gaming PC and doesn’t pretend to be one; it’s a developer appliance whose entire reason to exist is AI.
The unified memory is the part that changes the economics of local inference. On a conventional PC you juggle system RAM and VRAM as two separate budgets with a PCIe tax between them; here the CPU and GPU draw from one coherent pool, so a 176 GB model checkpoint simply exists in memory the way a spreadsheet exists in RAM — no offloading choreography, no layer-by-layer streaming. One box, however, is not enough for this model once you want a million tokens of context on top of the weights. Two boxes, joined by that QSFP cable, give you 256 GB of coherent-per-node memory and a private 200-gigabit fabric between them.
And the QSFP link is more interesting than it looks. On the GB10, one QSFP port is internally wired as two separate PCIe Gen5 x4 links — the Linux system sees two network interfaces and two RDMA-capable (“RoCE”) devices per port. That means a single cable between two boxes actually provides two parallel rails for the traffic between the model’s two halves, and an all-gather — the collective operation that stitches the two halves’ work together — runs about 1.8× faster on both rails than on one. Details like this are scattered all over the recipe, and they’re the difference between a demo and an appliance.
The model, and why a million tokens is even on the menu
GLM-5.3-Flash is the first natively multimodal release in Z.AI’s GLM-5 series: 320 billion total parameters with just 18 billion active per token, trained on a 30-trillion-token multimodal corpus, released under an MIT licence, with a technical report titled — with some self-awareness — “GLM-5: from Vibe Coding to Agentic Engineering.” Z.AI’s own claim is that it approaches Claude Opus 4.8 on coding and agentic benchmarks at a tenth of the price. Two architectural decisions matter enormously for serving it at home.
First, it’s a mixture of experts. Instead of one dense 320-billion-parameter brain, the model contains a large toolbox of “expert” feed-forward networks of which only a small, learned subset fires for any given token — 18 billion parameters’ worth. That’s why a 4-bit build of the full model weighs 176 GB yet decodes at 60-plus tokens per second on hardware this size: the memory is mostly dormant experts, and only the active slice does work per token. It’s also, as we’ll see, why quantizing this particular model turned out to be uniquely tricky.
Second, it uses a hybrid attention design: some layers use sparse attention with a compressed latent key-value cache, others use linear attention with tiny recurrent states. The practical effect is that the cache for a million-token context stops being a fantasy. A naive transformer’s KV cache for a million tokens would eat hundreds of gigabytes; GLM-5.3-Flash’s compressed latent cache plus its pooled sparse-indexer keys plus a few recurrent states fit in a handful of gigabytes. Z.AI layered Manifold-Constrained Hyper-Connections (a technique for getting more out of each layer’s representations) on top, trained the whole thing on that 30-trillion-token corpus, and shipped a 1,048,576-token window as the model’s native maximum. On a datacentre GPU with HBM to spare, that’s a footnote. On two desk boxes with 256 GB between them, it’s the entire point — and it’s why the community’s obsession with KV-cache bytes was justified rather than neurotic.
There’s a third thing worth knowing, because it shaped the whole saga: the model is agentic by design. It’s built for long tool-calling workflows — the card’s own evaluation notes read like a DevOps runbook (Terminal-Bench in Claude Code, 400k-context SWE runs, six-hour timeouts). Its chat template has a thinking budget (reasoning_effort: low, high, max), and the default is max. A model tuned to think hard for hours about a repository is a different beast to serve than a chatbot, and several of the bugs the community found — we’ll get to the reasoning-decay one — only existed in that agentic regime.
Act I: excitement (26–27 August)
When the model dropped on 26 August, the DGX Spark forum thread lit up within minutes. “Better than GLM-5.2,” announced the opener. Someone observed that the gap between frontier models and what you can run at home had collapsed from two years to under two months. The first community quantization — NVFP4, 195 GB, vision and multi-token prediction included, “first try” — appeared within the hour, and the quantizers never really stopped after that: RedHat, Intel, and the prolific local-inference-lab (“Luke”, who the thread anointed its gold-standard quantizer) all shipped versions in the first week. vLLM merged day-one support for the architecture; SGLang published a cookbook; even the tiny-model crowd got a 2.05-bit build that runs on a single Spark.
The first careful two-box benchmark landed the next day, and it’s worth reading as a snapshot — because it’s also a map of every problem the next month would solve. The setup: vLLM with a native Blackwell (SM120) kernel extension, a Marlin backend for the routed experts, an FP8 KV cache, and the model’s own MTP head drafting five tokens ahead. The progression tells the story in three lines: the baseline extension managed 7.96 tok/s; the native kernels lifted it to 14.5; adding speculative decoding took it to 23.1 — with aggregate throughput across five concurrent requests reaching 60.7. The benchmark author’s conclusion was honest: “the earlier 20–25 tok/s estimate was accurate.” (That estimate belonged to the thread’s resident pessimist, who had predicted the whole endeavour would end with a model that was “technically fit but nobody is using it”. He closed his post with an apology for being a party pooper. He was, at that moment, entirely right — and, as we’ll see, ultimately wrong in the most productive way.)
The same post catalogued the state of the art with unusual care: roughly 849,000 logical cache tokens with the drafter enabled — enough for four full 200k conversations, not five — an actual 180,013-token prompt passing end to end, and a table of bugs ranging from fluent gibberish (native MoE kernels corrupting output) to multimodal requests failing with “at most 0 images”. And the first quality score on the community’s brand-new agentic benchmark: 86/100. The model worked. It just wasn’t good yet.
One post the same week hinted that the pessimists were missing something: 99% on GSM8K, a maths score that beat the official DeepSeek release the forum kept comparing GLM against — 250 problems, greedy decoding, no tricks, from an early 4-bit build. A first hint that the quality story was going to be complicated. Nobody paid much attention yet; everyone was busy with the memory arithmetic.
Act II: the hangover (28 August – 9 September)
Then reality checked in, one disappointment at a time.
- Quants that lied about their quality. The thread adopted a rigorous benchmark — tool-eval-bench, a 69-scenario suite (plus 28 adversarial “Hard Mode” ones) grading tool selection, parameter precision, multi-step chains, refusals, error recovery, localization, and prompt-injection defence, with reproducible seeds and configuration fingerprints so two scores are never compared unless they’re comparable. Against it, one rushed NVFP4 release — attention heads requantized to squeeze in more speed — scored a disastrous 81/100 and was described, memorably, as “braindead”. It was pulled, fixed, and re-released within a day, but not before a sharp argument about why it had been pinned in a popular recipe. The episode taught everyone to demand measured fidelity instead of vibes.
- Silent corruption. CUDA graphs combined with speculative decoding quietly wrecked output quality; native FP4 MoE kernels produced syntactically valid nonsense; one checkpoint family leaked Mandarin and Russian into English replies. Each was hunted down with isolated kernel tests and fixed upstream, but only after people burned evenings believing their hardware was haunted.
- An out-of-memory that took a week to explain. In every two-box setup, the head node died at ~88% of the weight load while the worker sailed through — drawing 50 GB more memory for no obvious reason. The eventual diagnosis was beautiful: streaming the checkpoint from disk was polluting the Linux page cache, and the “missing” memory was cached weight bytes the system refused to reclaim. The fix was a pipelined instanttensor loader that reads weights with direct I/O — which as a side effect cut load time to about 50 seconds for 82 GiB. The same investigation produced a warning now pinned in every recipe: never put Docker memory limits on a GB10’s unified memory — the machine wedges, hard.
- Smaller gremlins. The stock chat template ignored the thinking toggle entirely (the model always thought, burning tokens). Long prompts stuttered to 0.08 tok/s at 40% of the window. The best drafter, DFlash2, shipped under a non-commercial licence. A degenerate loop made the model emit “!!!” forever. And the head-scratcher of them all: reasoning decay — past ~300,000 tokens of agentic tool use, the model silently stopped thinking and started emitting raw tool-call syntax. More on that in Act V.
For a fortnight the honest verdict was: brilliant model, unserveable hardware. One contributor’s first real agent workload ran 2× slower than DeepSeek, dropped 2 of 4 concurrent agents mid-task, and logged the whole failure. Several people went back to four-box clusters running other models at half precision, waiting for better quants.
It’s worth pausing on why this model was so punishing to quantize, because it explains the entire next act — and because the community’s answer to it became the template for how the model is served today.
Quantization works by spreading error. A dense 70-billion-parameter model has 70 billion chances to distribute the small inaccuracies that 4-bit weights introduce, so no single token’s computation is wrong by much. A mixture-of-experts with 18 billion active parameters has far less redundancy: fewer weights contribute to each token, so each one matters more, and GLM-5.3-Flash’s hybrid attention means its caches and sparse indexer do things that generic quantization recipes were never calibrated for. The first-generation quants proved it: benchmark scores that looked fine while the underlying token distributions had drifted meaningfully from the reference.
The community’s answer was to measure fidelity the proper way. The gold standard became KL divergence — a measure of how far the quantized model’s next-token probability distribution drifts from Z.AI’s own FP8 release — computed on the exact tokens the serving stack sees, in float32, with block-bootstrap confidence intervals. Suddenly “it feels fine” had a number attached, and the number exposed two things: that the early quants left real accuracy on the floor, and that part of the error came not from the experts at all but from the dense layers, which the TensorFold engine quantizes itself at load time. There was even a measured floor — the best achievable error with FP8 experts and 4-bit dense weights — so everyone knew exactly how close each new build had come to the physical limit. When a later quant arrived claiming improvement, the thread could check its work. That discipline is the difference between a hype cycle and an engineering programme.
Act III: the second front (10–13 September)
Meanwhile, a second thread opened — and it’s where my own part of this story begins. On 10 September, a contributor started “Let’s optimize nvidia/GLM-5.3-Flash-NVFP4 for 2× DGX Spark” with a simple question: NVIDIA had now released their own NVFP4 quantization of the model. Could it serve a 1M-token context on a dual-Spark setup? His first results were startling in both directions: 94/100 on the hard-mode benchmark — the best score anyone had recorded at that point — but only 262k context and 743,000 cache tokens. The gap between “best quality” and “fits on two boxes” was now a memory-budget problem, and the thread spent the next month closing it.
The breakthrough came two days later: by tuning the Docker runtime configuration and swapping the in-checkpoint MTP drafter for the external DFlash2 one, the same checkpoint served the full 1M-token context — with the recipe, Dockerfile and setup guide published for everyone. That post is where my own involvement starts, and I’ll be honest about how it went. I took those instructions and built a start.sh to pull it all together; I hit the notorious head-node crash (the log ending, unhelpfully, in “GLHF! 🏖”); I chased it down to a mangled recipe that had shoved an SSH login into the RoCE address field; I worked around a driver regression by downgrading the kernel; and then, at 900,096 tokens of context, it launched. Another contributor did the startup arithmetic that decided the question for everyone else: the 1M configuration needed about 118 GiB of free RAM per node at launch — a number you could check before wasting an evening.
The thread then turned into a masterclass in memory arithmetic. A precise failure analysis showed why NVIDIA’s own W4A4-style checkpoint (weights and activations at 4 bits) was the wrong starting point: it gained nothing in decode speed — decode was limited by reading the weights, which are 4-bit in every build — but it couldn’t use its own MTP head past 512k context and needed four restart attempts to boot at all. On the GB10 specifically, the analysis concluded, 4-bit weights with 16-bit activations was the sane choice, because the Spark’s NVFP4 tensor-core support was limited; full W4A4 was a datacentre play. A separate perplexity league table — bits-per-byte on wikitext and non-English Wikipedia, lower is better — ranked every quantization honestly, and the worst of the first-generation NVFP4 builds was so far behind that its proudest champion publicly deleted it. Meanwhile the reference recipe was re-verified at 700k context, still 94/100, and published as a launcher anyone could run.
The whole episode set the pattern for the rest of the saga: somebody benchmarks, somebody diagnoses, somebody ships the fix, and the recipe everybody uses gets better — with the receipts attached.
Act IV: the great convergence (14–30 September)
The back half of September is when the two threads stopped being two conversations and became one engineering programme, with four workstreams running at once.
Performance. The single biggest lever was found on 22 September: the GPU’s display reservation — a couple of gigabytes of video memory that a headless box can never use — could be clawed back from the driver and mapped into the KV cache, ~277,000 extra tokens per rank, no increase in memory budget. (Credit publicly given to Emi Huang for the technique.) It was independently validated at 1.32× cache size at the full 1M context, pushed to 1.38M, and — because a good trick travels — noted to work for the EXL3 recipes too. Prefill became an arms race: 921 tok/s on day one, 1,643 by mid-September with a rebuilt vLLM 0.29 image, 2,129 with a community kernel build, 2,257 with the next generation, peaking near 2,800. Decode climbed from the 23 tok/s of Act I to 40-plus, then 53–55 with a hand-requantized “FAST” build. Someone documented a ten-million-token disk-backed KV cache. And the clusters grew: three Sparks (one hack: bump the attention head count from 64 to 66 so it divides by three), four-box recipes at 90 tok/s decode with 3,500 tok/s prefill, and an eight-box reference.
Fidelity. Mia’s AI Lab published their EXL3 4-bit build with the twist already described: a full KL-divergence study against Z.AI’s FP8 release, cutting the quantization error by 11–27% at identical size and speed, with the “confident mistakes” that actually change answers reduced by 16–37%. A controlled A/B on identical hardware put the two formats within a point of each other on average — “identical ceiling, different floor”. And the thread’s most-cited quality study landed: the same benchmark suite scored across a 2.5× temperature range, showing NVIDIA’s own NVFP4 was not just the highest-scoring build but the tightest (a 1.3-point spread where competitors wandered by 5–7 points). Temperature invariance — the statistical signature of a well-calibrated quant — became the metric that separated the serious builds from the lucky ones.
Deployability. The gremlins got named and fixed. The “braindead” quant’s replacement shipped. A subtle bug where DFlash2 silently disabled prefix caching (re-reading a 5,600-token prompt from scratch on every turn) was diagnosed and fixed with a single retention-interval flag — verified by watching the second identical prompt return in a fraction of the time. The chat-template thinking gate got fixed in every template. The out-of-memory gremlin from Act II never came back.
And quality kept sneaking up on everyone. Hard-mode scores of 97, 97 and 95 appeared from one setup across three consecutive runs — the kind of consistency that made a sceptic (me) finally ask on the forum whether the early 90s had been “consistent or a fluke”, and joke that the benchmark needed a --hardhardmode flag. The answer, from the thread’s benchmark farmer, was a methodology: score everything, multiple times, at fixed seeds, and report the mean — because “it’s about consistent mean, not top flukes.”
Act V: the reasoning revelation (late September – 8 October)
Somewhere in here, the conversation shifted from how fast to how good — and this is the part I didn’t expect. The anecdotes started arriving from people using the two-box setup as their daily driver, not as a benchmark rig. One contributor described running their hardest work projects on GLM on two Sparks, fast jobs on a smaller model on a Mac, and only the very hardest problems to a cloud frontier model — a three-tier desk strategy, with GLM holding the middle that used to require the cloud. Another reported 53–55 tok/s from a hand-optimized build “with quality unaffected”, and set about retraining a drafter properly. Another’s verdict after weeks of production use: speed now “on par with DeepSeek-V4-Vision, but its quality is a cut or two above.”
The benchmarks kept pace. 96/100 on the hard-mode suite — reproduced with zero variance across three trials, which in benchmark terms is eerie. A contributor fed it 3.5 years of legacy .NET and had it port to Python overnight at a 99.7% trace match. LiveCodeBench put a TensorFold build at 91% against a hosted frontier endpoint’s 93% — “not dumbed down”, in the tester’s words. And the anecdote that travelled furthest: a user’s production ERP patch workflow moved onto the two-box setup, unattended, at full trust, displacing a Claude subscription entirely. The model the sceptics assumed would be lobotomised by 4-bit quantization turned out — with the right recipe — to reason like the frontier models it was undercutting on price by roughly 100%.
The community even turned the quality question into science. A Cyrillic long-context study swept 8k to 480k tokens across every quantization and found a precise failure signature — facts degrade first, code and needle retrieval stay intact, the onset depends on the quantization (FP8 holds to ~384k tokens, NVFP4 to ~192k, one build never degrades at all) — and, beautifully, that with thinking mode enabled the degradation vanishes entirely. “Quantization quality manifests itself precisely in thinking-off mode.” That’s the kind of finding you usually need a paper for; this one came with reproducible scripts and a benchmark repository.
It’s worth being precise about what the model is, per Z.AI’s own card: it approaches Claude Opus 4.8 on coding and agentic benchmarks. Not equals. The thread’s final quality rankings put the best two-box stacks within touching distance of the FP8 reference on coding — the residual gap showing up in exactly the places quantization always bites: the most confident 1% of decisions, the deepest contexts, the longest agentic loops. What you get for the compromise is a model that runs on a shelf. Ask anyone who has paid per token for agentic workflows which side of that trade they’d rather be on.
Act VI: the resolutions (8–12 October)
Every bug that made Act II painful now has a documented resolution. The reasoning-decay bug — the model silently stops thinking after ~300k tokens of agentic tool use, collapsing into raw tool-call syntax — turned out to have a precise mechanism: turns whose conversation tail ends with a tool result rather than a user message were being served without the reasoning scaffold. It was diagnosed, reproduced deterministically, fixed by restoring the thinking scaffold after tool calls, and validated to 774k tokens across 100 rounds. The degenerate loops got a loop guard plus upstream fixes. The out-of-memory got the pipelined loader, and the disk spill tier that makes long conversations instant to resume — one build demonstrated a ten-million-token spill. The French-prompt end-of-turn bug got DENSE=fp8 and a measured explanation. The thinking-gate got fixed in every chat template. Even the cache got an operating manual, after one incident report showed a server grinding to a halt with a 99%-full cache: keep your agents inside 70–80% of the pool and the eviction storms never happen.
And the thread closed with a consolidated fourteen-configuration benchmark matrix — a map of the entire quality/speed trade-space, published for anyone to follow — plus a new deterministic agentic-coding benchmark so the next model doesn’t have to rely on vibes. Its final post delivered the verdict I’m still smiling about: the author was now running, in parallel, “gilesco’s stack (TensorFold 0.6.4) with applied Mia’s latest PR’s” and technigma’s latest vLLM stack — “both have best overall performance and stability.” Gilesco’s stack. Mine. The start script born from a crash log in September had become, by October, one of the two reference configurations for this model on two boxes — and my EdgeXperts had quietly graduated from “can it even boot” to “co-champion, by someone else’s benchmark”.
The state of the art, as of this week
The final consolidated matrix reads like a menu of trade-offs rather than a winner’s podium:
| Stack (2-box, tensor parallel) | Hard-mode quality | Prefill | Decode (c1) | KV pool | Long-context fidelity |
|---|---|---|---|---|---|
| Native NVFP4 stack (quality crown) | 92 / 96 max | 2,678 tok/s | 42.7 tok/s | 1.22M tokens | degrades past ~192k (thinking on: no) |
TensorFold EXL3, DENSE=fp8 (speed & pool king — what I run) |
91 / 93 | 1,601 tok/s | 52.7 tok/s | 2.29M tokens | degrades past ~384k (thinking on: no) |
| W4A16 purist build (the perfectionist’s choice) | 91 / 92.3 | 1,510 tok/s | 25.4 tok/s | 1.34M tokens | zero degradation at any depth |
| DeepSeek-V4-Flash (reference cloud model) | 89.7 / 89 | 1,746 tok/s | 36.9 tok/s | 3.62M tokens | — |
There is no single “best”. There’s a menu — and since we’ve earned it, here it is properly.
The tasting menu
Every dish below is served on two GB10 boxes with a QSFP cable between them. Reservations not required.
🍸 Amuse-bouche — Single Spark, 2.05-bit
One box, one absurdly brave quantization. 64 tok/s decode, which is genuinely faster than some two-box builds from September — but the context window and the reasoning depth pay for it. An amuse-bouche, not a meal. Pairs well with curiosity and a backup plan.
🥂 The Official Pour — NVIDIA NVFP4 + DFlash2
The house reference: NVIDIA’s own quantization, the external drafter, tuned over RoCE with CUDA graphs and async scheduling. Verified to 700k context at 94/100 hard-mode, the tightest temperature stability of any build, and the full million-token window. Strengths: fidelity, stability, the recipe everyone can reproduce. Weaknesses: wants 118 GiB of free RAM per node at launch, and the drafter’s licence is non-commercial. Pairs well with anyone who wants the vendor’s own checkpoint and a forum thread’s worth of collective tuning behind it.
🏆 The Crown Jewel — The native NVFP4 stack
The quality leader: 96/100 hard-mode with zero variance across three trials, fastest prefill of any two-box build (2,678 tok/s), the reasoning-decay fix, and thinking restored after tool calls. Strengths: the last word in quality; where the deepest agentic workflows belong. Weaknesses: the smallest KV pool of the premium builds (1.22M tokens) — it likes a little headroom and dislikes being crowded. Pairs well with long, hard problems and short guest lists.
🔥 The House Special — TensorFold EXL3, DENSE=fp8 (what I run)
The speed and pool king: 61.8 tok/s single-stream, 105.5 aggregate at four concurrent (128 in code), ~2.9M tokens of shared KV pool, a disk spill tier that resumes a 200k conversation in 0.6 seconds, and the ten-million-token spill demonstrated. Strengths: throughput, pool size, the best points-per-benchmark-hour in the consolidated matrix, and the quietest fans. Weaknesses: the quality floor sits a point or two below the crown jewel at the very deepest contexts, and the 4-bit dense layers want DENSE=fp8 if your prompts are French. Pairs well with households of four, million-token codebases, and people who like their inference like their coffee: fast and local.
🧘 The Ascetic — The W4A16 purist build
Every accuracy-critical weight left in BF16, experts at 16-bit activations. The only build measured with zero long-context degradation at any depth — 212 of 212 scenarios, no cuts, no loops, at 8k or 480k tokens. Strengths: purity; the choice of document lawyers and anyone whose prompt is a quarter-million tokens of contract. Weaknesses: 25.4 tok/s — half the House Special — and a pool that won’t impress anyone. Pairs well with patience, precision, and readers who skipped the puns.
🏰 The Banquet — The 4× and 8× Spark towers
Tensor parallel across a cluster: 90 tok/s decode and 3,500 tok/s prefill on four boxes; reference scores on eight. Strengths: everything, in bulk. Weaknesses: the cable bill, the power bill, and the moment you have to explain the cabling topology to a guest. Pairs well with small teams and large ambitions.
🫧 Off the menu — the Mac Studio over MCDMA
An independent experiment extended the two-box setup with an M3 Ultra over a third networking path. It worked — 61.3 tok/s, right in the neighbourhood — but end-to-end latency came in slower than the plain Ethernet-class fabric. The community documented it as a reproducible result and not a recommendation, which is exactly the culture you want. Not served. Documented with love.
The cast
Threads like these are usually remembered by their drama; this one deserves to be remembered by its roll call. A partial cast, and what each brought:
| Who | What they brought | Why it mattered |
|---|---|---|
| Z.AI | The model: hybrid sparse/linear attention, a 1M window, MIT licence, day-one multi-framework support | Made the whole endeavour possible; the 1M window is why two desk boxes are interesting |
| NVIDIA | The official NVFP4 checkpoint — and, implicitly, the memory-budget puzzle of serving it on two boxes | Started as the quality crown and 262k context; ended as the foundation of several 1M recipes |
| PILCOTHINK | The optimization thread itself, the 1M-context NVFP4 breakthrough, the vLLM 0.28→0.30 Docker images | My start script descends directly from his instructions; prefill went 921 → 1,744 tok/s through his images |
| ashhart | TensorFold — the engine the recipe runs on, with exactness guarantees (drafted replies must equal serial ones) | Two-box serving with correctness as a design goal, not an afterthought |
| Mia’s AI Lab | The EXL3 4bpw checkpoint, KL-divergence-calibrated; the TensorFold recipe; the disk spill tier | Turned “technically fits” into “measurably faithful”, then “fast” |
| SeraphimSerapis | tool-eval-bench: the 97-scenario hard-mode benchmark with configuration fingerprints | Replaced vibes with a ruler; every score in this story is comparable because of it |
| 0rand | Day-one scepticism, then the reference NVFP4 launcher, the display-reservation trick, the temperature-invariance study, the reasoning-decay discovery, and a deterministic agentic benchmark | The thread’s conscience and its most prolific troubleshooter |
| ajvazan | The benchmark farm: fourteen consolidated configurations, the depth-degradation methodology, the sparkring teardown | Turned anecdotes into data; the thread’s statistical backbone |
| technigma.ai | The quality-crown stack, the display-reservation and reasoning-decay fixes | Proof that the last 5 points of quality were findable |
| helge & protryt | The page-cache OOM diagnosis and the instanttensor loader | Fixed the single most common crash in every two-box setup |
| vedcsolution | Load-time quantization of the checkpoint’s BF16 attention projections and shared experts | +23–27% decode from a pull request, by reading what NVIDIA left unquantized |
| jetspark | The perplexity league table, the W4A4 memory post-mortem, the drafter-acceptance analysis | Killed a bad quant, crowned a good one, and proved which drafter length actually helps |
| local-inference-lab, RedHat, Intel, canada-quant | The quantization ladder | Each rung raised quality, size efficiency, or licence freedom |
| wojo, florianbrede, BadAd84, lukaszraczylo, eduffd and more | The spill tier, the FP8 dense findings, long-context fixes, the Anthropic API backport, and roughly a hundred other patches | The long tail that turned a demo into infrastructure |
| giles8 | A start script built from a crash log; a stubborn question about whether 97/100 was “consistent or a fluke”; and eventually the TensorFold rebase the thread ended up recommending | That’s my run. Hi. |
By the numbers
My two EdgeXperts run the TensorFold EXL3 recipe, with GPU clocks deliberately capped at 2,200 MHz — the quiet-and-cool configuration, not a benchmark-chasing one:
| Metric | Result |
|---|---|
| Decode, 1 prose stream | 61.8 tok/s (TTFT 162 ms) |
| Decode, 4 concurrent (aggregate) | 105.5 tok/s |
| Decode, 4 concurrent (code) | 128.3 tok/s |
| Prefill throughput | ~2,000 tok/s up to 131k tokens |
| First token on a 131k prompt | ~69 s |
| Tool calling (97-scenario hard mode) | 91/100 |
| Context / shared KV pool | 1,048,576 tokens / ~2.9M tokens |
A note on what those numbers feel like from the outside. A 131k-token prompt — a small book, or a fat stack of source code — takes about a minute to read before the first word of the answer. After that, replies stream faster than you can read them. Four people can use the box at once without noticing each other, and a conversation can sit for days and resume mid-thought in under a second. It is, in the most literal sense, a service — just one whose data centre is 1.2 kilograms and whose licence fees are zero.
Why fork it?
This is my fork of MiaAI-Lab’s original recipe — with the note that all of my commits since the fork were generated with AI assistance. It’s been a pleasant dogfooding loop: a model served from a repository it helped to write, benchmarked in the thread’s final week, and now writing blog posts about itself.
Full instructions, benchmarks and the configuration reference are in the README, and the whole saga — every dead end, drama, bug and benchmark — lives in the two forum threads (here and here). The shopping list is short: two GB10 machines (DGX Sparks or EdgeXperts — the recipe doesn’t care whose logo is on the case; a QSFP cable even ships in the box on some SKUs), and about 205 GB of disk on each. What you get in return is a frontier-class assistant with a million-token memory that never phones home, and whose entire supply chain — weights, engine, patches, benchmarks, and this story — was built in the open by people arguing constructively on a forum. Long may it continue.









