Native Mac inference for agents at scale
inferd is a native inference engine for Apple Silicon, built for many agents at once: long shared contexts, agents that pause and come back, and everyone wanting an answer now.
01 · The problem
Most Mac engines are tuned for one person talking to one model, and they are very good at it. Agents behave differently: they share long contexts, take turns, pause, resume, and all want an answer at once.
One prompt at a time
When prompts are read strictly one after another, the queue grows with every agent. In one engine's parallel mode, combined throughput even falls as agents are added: 148 → 143 → 124 tok/s at 2, 4 and 8.
inferd: everyone, together
Prompts are read in chunks between decoding steps, agents are admitted by memory budget, and the GPU stays busy. Add agents and the combined rate keeps climbing.
02 · Measured on a Mac Studio
Same machine, same weights, same harness. Every point on every chart is a measured run, not an estimate.
inferd is not ahead everywhere, and it stands on excellent work. These are the engines we measure ourselves against, and where they lead.
TensorFold 1.2.1 finishes sooner up to about 5 agents (its Flash-Next prefill is faster); inferd from 6 (8 agents: 137.7 vs 166.3 s). Seconds, 3 trials.
03 · Pick your Mac
Every figure below was measured on that machine, not extrapolated.
Small box, serious work. On every 8- to 64-agent fleet we ran on the minis, inferd led.
| Qwen3.8-27B 4-bit | M4 Pro 64 GB | M6 32 GB | M5 Pro 64 GB |
|---|---|---|---|
| Decode, one agent, 4k | 31.8 | 37.2 | 72.6 |
| Prefill, 4k prompt (PF8 off) | 120 | 269 | 454 |
| 16 agents, 4k contexts | 65.4 | 110.5 | 184.6 |
| 16 agents, 16k contexts | 59.6 | 67.6 | 167.6 |
| Resume via the session API | – | 0.43 s | 0.26 s |
| Self-tests and demos | 15/15 pass | 15/15 pass | 15/15 pass |
tok/s unless stated. With int8 prefill (PF8) on a server sized for a few agents, the M5 Pro reads a 4k prompt at 668 tok/s.
Flash-Next does not fit on a mini. For the biggest models, step up to the Studio.
256 GB of unified memory holds dozens of long contexts, Flash-Next and an int8 prefill copy of the weights at once.
| Qwen3.8-27B, 32 agents | 16k | 32k | 64k |
|---|---|---|---|
| 4-bit, total tok/s | 615.2 | 576.3 | 434.8 |
| 4-bit, Splash 1.5 | – | – | 310.6 |
| 8-bit, total tok/s (1 Nov build) | 599 | – | – |
| All 32 agents done at 64k | 37.7 s (Splash 52.7 s) |
Follow-up turns with 512-token replies, 0 errors.
Runs on open weights: Qwen3.8-27B · Qwen3.8-Flash-Next · Qwen3.6-35B-A3B · Qwen3.5-9B · Gemma-4-31B · Muse Glimmer 30B
04 · Built for agents
Agents re-read the same material constantly. inferd lets the agent framework, which knows who will come back, decide what the model's memory keeps.
One field on a normal OpenAI request
POST /v1/contexts # save + pin a context once POST /v1/chat/completions { "model": "qwen3.8-27b", "messages": [ ... ], "inferd": { "context": "brief-42", # start from it "session": "agent-7", # keep my turn "effort": "low", # per agent "max_context": 32768, "priority": 1 } } # one-off work that leaves no trace: "inferd": { "cache": "drop" } GET /v1/cache # cached, pinned, parked, free
Every reply reports cached_tokens, so a framework can check its caching strategy instead of guessing.
05 · Quality
06 · On the shoulders of giants
No Python on the token path; a thin Python front end serves the OpenAI-compatible API. inferd builds on Apple's MLX (its quantized and attention kernels are vendored, MIT), Qwen's open weights, the DFlash speculative drafter, and lessons openly shared by the TensorFold, Splash, oMLX and vLLM communities. Built and tested on Mac Studio M5 Ultra, Mac mini M6 / M5 Pro / M4 Pro and MacBook Pro M5 Max, macOS 27.
Tell us your models, your Mac and your workload, and we'll show you where inferd fits. The full report, harness and raw results are available on request.