sirfyynProject · released October 2026 · MIT
FLE — Fast Local Experts
An inference engine for large Mixture-of-Experts models on one graphics card. Built as a fork of llama.cpp, every change backed by paired measurements — including the ones that didn't work.
GitHub Technical report Model files
- Decode
- ~90 tok/s
- FLE-IQ3_M
- 88.9 GB
- GPU
- 32 GB
How it works
- Expert cache in VRAM. The most frequent experts stay on the GPU; admission by frequency, oldest-first eviction.
- GPU and CPU at once. Cache hits run on the GPU while the CPU computes the missing experts from system RAM; plus fused CUDA kernels.
- The model's own MTP head. It drafts two tokens; the main model checks them in its next pass; a 32k draft vocabulary makes drafting cheaper.
- Lean prefill. Long prompts borrow cache layers for larger batches, plus a prompt cache that doesn't evict follow-up turns.
- Compatible. Same GGUF format and server API as llama.cpp.
Specs
| Decode speedFLE-IQ3_M, RTX PRO 4500 32 GB; 91.2 tok/s on six tasks | ~90 tok/s |
|---|---|
| Against llama.cppsame file, same GPU, same 16 threads | 2.49× |
| Against the previous setupstock llama.cpp with Unsloth UD-IQ4_XS, ~56 tok/s | +58.5 % |
| Against Strata91.2 vs 74.0 tok/s on six tasks, not paired | +23 % |
| Prefillllama.cpp with -ub 4096 is 24 % and 12 % faster at 4k and 8.6k tokens; the next work item | slower |
| Model size FLE-IQ3_MUnsloth UD-IQ4_XS: 93.7 GB | 88.87 GB |
Measured on “Titan”: AMD Ryzen 9 9950X3D (16 cores), 120 GB DDR5, NVIDIA RTX PRO 4500 (32 GB). All numbers except the Strata comparison come from paired measurement brackets with 95 % intervals; the details are in the technical report.
What it bought
Every change ran in a paired measurement: baseline, candidate, baseline, on the same machine. Two ideas made it slower and were dropped.
| Expert cache + admission/eviction | +12.2 % | |
|---|---|---|
| Overlap + CUDA fusion + IQ4_NL head | +19.5 % | |
| Re-trained MTP head (not released) | +4.9 % | |
| 32k vocabulary list (draft head) | +4.1 % | |
| 46 instead of 44 cached layers | +2.3 % | |
| Prefetch experts (pre-gating)dropped | −10 … −27 % | |
| N-gram draft instead of MTPdropped | −14 … −23 % |
Change in decode speed against the respective baseline. Effects build on each other.
FLE-IQ3_M: smaller, just as good
FLE-IQ3_M (internally “G3”) combines the Unsloth UD-IQ4_XS base with expert layers from ISTA-DASLab GSQ-RCO and saves almost 5 GB. Its KL divergence against the original differs only within measurement noise (±0.007) on three texts; one of them was held out during selection.
| Text Fselection text | ΔKLD −0.0077 |
|---|---|
| Text Echeck text | ΔKLD +0.0029 |
| Text Vheld out | ΔKLD −0.0026 |
What is released
- FastLocalExperts on GitHub — the engine (MIT), version 1.1.0 (releases, portable CUDA builds for Linux and Windows since v1.1.0), with the launch command and every FLE switch.
- FLE Studio — a local app with chat, live data, model loader and launch profiles for FLE and llama.cpp (MIT, version 0.3.0; GitHub).
- Model files on Hugging Face — FLE-IQ3_M, FLE-IQ3_XXS, FLE-IQ2_M and the MTP head, with per-layer provenance; the REAP-pruned version has its own repo.
- The technical report — every measurement, including failures; archived on Zenodo (doi:10.5281/zenodo.23272400).
- Eight short films — “Making Rhea fast”, seven minutes on YouTube (channel @sirfyyn), how FLE was measured and what did not work.
Sources
- llama.cpp / ggml — G. Gerganov and contributors, MIT license
- Qwen3.8-Next — Qwen Team, “On the Design of Qwen3.8-Next Architecture”, arXiv:2608.30320
- Unsloth Dynamic GGUF — base quant UD-IQ4_XS
- ISTA-DASLab GSQ-RCO — donor of the expert layers; GSQ arXiv:2604.18556, RCO arXiv:2605.00649
- imatrix & IQ-Quants — I. Kawrakow, llama.cpp #4861, #4897, #5676
- KLD as a metric — Dutta et al., “Accuracy is Not All You Need”, arXiv:2407.09141
- Multi-Token-Prediction — DeepSeek-V3, arXiv:2412.19437; Gloeckle et al., arXiv:2404.19737
- Speculative decoding — Leviathan et al., arXiv:2211.17192; Chen et al., arXiv:2302.01318
- Hyper-Connections — Zhu et al., arXiv:2409.19606
- MoE offloading — Fiddler arXiv:2402.07033; HybriMoE arXiv:2504.05897; MoE-Infinity arXiv:2401.14361
FAQ
Can Qwen3.8-Flash-Next run on a single 32 GB GPU?
Yes. FLE puts the dense layers and a cache of frequently used experts on the GPU; the remaining experts stay in system RAM and are computed on the CPU. On an RTX PRO 4500 with 32 GB I measured around 90 tokens per second for decoding.
Is FLE a fork of llama.cpp?
Yes. FLE is a fork of llama.cpp (MIT license) and keeps its GGUF format, quant types and server. All FLE features are opt-in; without them it behaves like llama.cpp.
What is FLE-IQ3_M?
The model file behind the numbers (internally “G3”): a hybrid quant of Qwen3.8-Flash-Next from Unsloth UD-IQ4_XS and ISTA-DASLab GSQ-RCO, 88.87 instead of 93.7 GB, with a KL divergence within measurement noise.
Where can I get FLE?
Public since October 2026: the engine on GitHub (sirfyyn/FastLocalExperts, MIT), the model files on Hugging Face (sirfyyn/Qwen3.8-Flash-Next-FLE-GGUF) and the technical report at sirfyyn.github.io/FastLocalExperts, archived on Zenodo (doi:10.5281/zenodo.23272400).
Is there a graphical app for FLE?
Yes: FLE Studio is a local app for macOS, Linux and Windows with a chat in your browser, live data while the model answers, a model loader and launch profiles. It works with FLE and with plain llama.cpp, with no account and no telemetry (MIT).