How it works
Pick a mechanism to watch it work, and the measurement that backs it. Each one is something the engine does so that dozens of agents can share a Mac.
Data-centre GPUsServing many requests at once is standard, where engines like vLLM do it for cloud AI.
→Mac toolsGrew up serving one person and one conversation, and do that very well.
→inferdDoing it for dozens of agents means scheduling, memory and caching built for sharing, and that is what it adds.
continuous batching
Every active agent gets its next word in the same step.
The model writes one token (about three-quarters of a word) at a time. Instead of finishing one agent's reply before starting the next, each step of the GPU writes the next token for every active agent together. New agents join between steps, so nobody waits for someone else's answer to end.
MeasuredThroughput as agents are added →
chunked prefill
A long prompt is read in slices, so a new arrival freezes no one.
Before an agent can reply, the model must read its prompt, which can be a whole codebase. inferd reads long prompts in chunks between those steps, so a new arrival does not freeze everyone else. Engines that read prompts strictly one after another make the queue grow with every agent; in one engine's parallel mode the combined rate even fell as agents were added (148 → 143 → 124 tok/s at 2, 4 and 8).
paged, 8-bit KV cache
Twice as many agents' notes fit in the same memory.
While it works, the model keeps notes on everything it has read for each agent. inferd stores those notes in small pages at 8 bits instead of 16, so twice as many fit, and memory goes wherever an agent needs it. Agents are admitted by memory budget; if memory runs out, the newest one pauses and resumes later.
MeasuredThroughput on every Mac →
prefix cache and state snapshots
A shared brief is read once and reused by every agent.
Agents often share a long starting point: the same brief, documents or system prompt. inferd keeps what has already been read and lets every agent start from it. Qwen3.8 models mix two kinds of layers, and one of them keeps a running summary rather than notes; inferd snapshots that summary too, which is what makes this work on these models.
sessions and the SSD tier
Back at work in 0.2–0.7 s, not 15–18 s.
An agent waiting on a tool or a person can be parked in memory or on the Mac's SSD, with checksums. When it returns it carries on in 0.2–0.7 s instead of re-reading everything (15–18 s).
MeasuredBack to work in a blink →
speculative decoding
The same answer, in fewer steps.
A small, fast draft model guesses the next several tokens; the big model checks all of them in a single step and keeps the ones it agrees with. The answer is the big model's either way, but it arrives in fewer steps.
MeasuredThroughput as agents are added →
native Metal, int8 compute
GPU, unified memory and Neural Engine all work on it.
inferd is written in C++ and Metal for Apple Silicon, with no Python in model execution. It uses the GPU's integer path (int8) where it is faster, Apple's unified memory so nothing is copied between CPU and GPU, and the Neural Engine (on by default for reading prompts on M4; optional elsewhere).
MeasuredThe same model on every Mac →
Architecture
No Python in model execution: the model, sampling and scheduling are native; a thin Python front end detokenizes, parses tool calls and serves the OpenAI-compatible API. inferd builds on Apple's MLX (its quantized and attention kernels are vendored, MIT), Qwen's open weights, the DFlash speculative drafter, and lessons openly shared by the TensorFold, Splash, oMLX and vLLM communities.
Keep going