Open source · Rust host · CUDA kernels · speaks vLLM's API
FastLLM is an LLM serving engine with a compiled, memory-safe Rust host and GPU kernels generated for your model and your accelerator. Point your vLLM clients at it and get more tokens per second, a lower time to first token and no Python in the hot path.
The scheduler, KV-cache pool, request I/O and step loop are compiled Rust. That rules out whole classes of
bugs (buffer overflows, use-after-free, data races) by construction, and it means no interpreter or
pickle loading in the serving path. One static binary with a small attack surface.
Generic kernels leave performance on the table. FastLLM searches kernel configurations for the exact model shapes and accelerator you run, gates every candidate on correctness and installs the winners as a kernel table that the engine loads at startup.
FastLLM speaks vLLM's wire protocol, so the OpenAI-compatible clients, routers and benchmarks you already run keep working. Change one URL and compare the two engines on your own traffic.
The host does about a thousand times less work per request than a Python scheduler, so the GPU never waits for it. The gap widens on the fastest GPUs: on H100 and B200 a Python host becomes the bottleneck first.
Every model × accelerator pair gets its own kernels.
Most engines ship one set of kernels and hope it fits. FastLLM treats tuning as a search problem. Give it a model and a GPU, and the optimizer explores tile shapes, split strategies, thread counts and fusion boundaries, discards any candidate that disagrees with the reference output, measures the survivors under the profiler and writes the fastest into the engine's kernel table.
Next: a learned model of the accelerator that predicts which kernel wins before it is measured, and then writes new kernels outright.
Users feel time to first token and the gap between tokens. A host that schedules in microseconds and a decode step with fewer launches show up directly as a snappier stream at every concurrency.
Agents resend long, shared prefixes and fire many short calls in a loop. Prefix caching reuses the shared KV-cache blocks across those calls, and the cheap host keeps hundreds of concurrent tool loops moving without stalling the GPU.
Offline generation for evaluation, data pipelines and the rollout phase of RL training runs at high concurrency, which is exactly where the step loop and kernel tuning pay off most.
Every number below says which model, shape, concurrency and device it came from. Kernel times are Nsight Systems on both engines. Ratios are vLLM time ÷ FastLLM time, so higher is better.
The H100 NVL served-throughput runs were on a confidential-computing host, which makes device-to-host copies blocking and magnifies host overhead for both engines. At long context (16k–32k tokens) both engines stream the KV cache at 85–90% of peak bandwidth, and the decode step is at parity. Full methodology and raw data live in the repository.
Run FastLLM next to your vLLM deployment, send it the same traffic and compare the results.