moomou

(ﾉ≧∇≦)ﾉ ﾐ ┸┸

Building mini_sglang from Scratch

Posted at — May 24, 2026

Overview #

I have been experimenting with SGLang and vLLM for hosting Qwen for local AI. While I felt comfortable running and benchmarking these systems, I did not really understand the internals of a production grade inference engine.

With coding agents, I set out to correct that by working through a series of LLM guided lessons to build a mini sglang from scratch; specifically, I implemented a minimal inference engine that serves Qwen3 with paged KV cache, continuous batching, streaming output and a radix prefix cache.

The LLM generated lessons along with some of my questions is at mini_sglang and the code is on GitHub. The rest of this blog provides a short walk-through and highlights that I found most interesting.

Background #

Stripped of optimizations, an LLM inference server repeatedly does the following:

  1. Tokenize the input text.
  2. Decide which waiting and running requests should be in the next batch.
  3. Reserve KV cache space for the tokens in that batch.
  4. Run the model and write the new keys and values into the cache.
  5. Sample one output token for each request.
  6. Incrementally turn those token IDs back into text.
  7. Stream the text to the clients and repeat until each request is finished.

The purpose of mini_sglang is to keep these pieces visible. It is single-GPU and does not implement tensor parallelism, quantization, speculative decoding or CUDA graphs. Those are important for a production engine, but they would have made it harder for me to see the basic request lifecycle.

Architecture #

The server is split across two threads. FastAPI runs on the asyncio event-loop thread while the scheduler and model run on a dedicated engine thread. This separation is necessary because scheduler.step() launches GPU work and blocks; running it directly inside an async request handler would also block every other HTTP request.

Architecture diagram showing a request crossing from FastAPI into the mini_sglang engine thread, through the scheduler and Qwen3, then streaming generated text back to the client

When a request arrives, the HTTP handler tokenizes the prompt, creates an output asyncio.Queue for that request and puts a GenRequest on a thread-safe pending queue. The engine thread drains the pending queue and gives the request to the scheduler.

For every step, the scheduler selects a mixture of waiting prefills and running decodes, reserves blocks and builds a ForwardMeta instance. The model only sees the input token IDs, the metadata and the shared KV pool; it does not need to know anything about HTTP requests. After sampling, the engine updates each request and sends decoded text back to the event loop with loop.call_soon_threadsafe. The FastAPI handler simply waits on its output queue and yields SSE chunks.

Keeping all scheduler and cache state on the engine thread also made the implementation much easier to reason about. The only objects crossing the thread boundary are new requests and generated text.

One Mini Step at a Time #

I organized the project so that each lesson adds one new idea:

Every lesson keeps the same basic contract: greedily generate 20 tokens for "The capital of France is" and compare the token IDs against transformers.AutoModelForCausalLM.generate. Having this small end-to-end test was useful because most of my bugs did not produce an exception. An off-by-one position or an incorrect cache slot simply caused the model to produce different text several tokens later.

Paged KV Cache #

The first implementation in L1 uses a contiguous KV tensor. This is fine for one request, but a server would have to reserve the maximum sequence length for every request and pad batches to the longest sequence.

With paging, the KV pool is divided into fixed-size blocks. If the block size is 16, a request with 73 tokens needs five blocks. The physical blocks do not need to be next to each other:

logical tokens:  0 .............. 72
request.blocks:  [7, 2, 19, 4, 11]

KvPool[layer].shape =
    [num_blocks, block_size, num_kv_heads, head_dim]

Request.blocks describes ownership while ForwardMeta describes the current model call. In particular, slot_mapping says where the new K and V tensors should be written, while block_table and seq_lens_kv tell attention where to read the existing history.

I initially expected the attention kernel to be the difficult part. Most of the work was actually keeping the allocator, request length, slot indices and metadata consistent. A common bug was to increment cur_len by one after every forward call. That is correct for decode, but a prefill of 32 tokens must advance it by 32.

Continuous Batching #

The scheduler rebuilds the batch before every model call, so a new prefill can run beside requests that are already decoding:

step 0:  prefill A (8 query tokens)
step 1:  decode A (1) + prefill B (32)
step 2:  decode A (1) + decode B (1) + prefill C (5)
step 3:  decode A (1) + decode B (1) + decode C (1)

The query tokens are packed into one flat tensor instead of padded into a conventional batch. Suppose three requests contribute 5, 1 and 3 query tokens:

q_lens = [5, 1, 3]
cu_seqlens_q = [0, 5, 6, 9]
last_logit_rows = [4, 5, 8]

The model returns logits with shape [9, vocab]. Rows 4, 5 and 8 are the final query positions for the three requests, so those are the rows sent to the sampler. Walking through this example made cu_seqlens_q much less mysterious to me. Continuous batching is possible because each request can have a different query length while sharing one forward call.

Incremental Detokenization #

One part I did not expect to spend much time on was turning generated token IDs back into streaming text. Decoding each token independently does not work because a multi-byte UTF-8 character can span several BPE tokens. Decoding the entire history after each token is correct but becomes O(N²).

The solution is to decode a small window and emit the difference from the previous window. The subtle case is when a token completes a character that previously decoded as one or more Unicode replacement characters. Checking only whether the current window ends in \ufffd is not enough:

decoded_window = tokenizer.decode(window, skip_special_tokens=False)
if decoded_window.endswith("\ufffd"):
    return ""

prefix_text = tokenizer.decode(window[:-1], skip_special_tokens=False)
if "\ufffd" in prefix_text:
    full_text = tokenizer.decode(tokens, skip_special_tokens=False)
    new_text = full_text[emitted_len:]
else:
    new_text = decoded_window[len(prefix_text):]

For Qwen3, the emoji 🫨 is split into three tokens. The first two pushes produce replacement characters and emit nothing. On the third token, the fallback above compares against the amount of text actually emitted and returns the emoji. Without it, the replacement characters make the prefix appear longer than the final decoded string and the emoji does not appear until the stream is flushed.

This ended up being one of the more useful debugging exercises in the project because the first implementation looked reasonable and passed ASCII-only tests.

Radix Prefix Cache #

The last lesson adds a radix tree keyed by token IDs. Chat requests often begin with the same system prompt, so recomputing its KV on every request is wasteful. A new request searches for the longest cached prefix and starts prefilling at the first unmatched token.

The tree itself was not the hard part. The difficult part was defining who owns a physical KV block. After adding the cache, a block may be referenced by both active requests and radix-tree nodes:

block refcount =
    active requests holding the block
    + radix nodes holding the block

A cache hit increments the refcount because the new request now shares the blocks. Finishing a request decrements its references, but the blocks stay alive if the radix tree still owns them. Eviction removes the tree’s references, and a block only returns to the free list when its refcount reaches zero.

The cache also stores only block-aligned prefixes. If two prompts share 97 tokens and the block size is 16, only the first 96 tokens can be reused. Rounding this value up is especially bad: the request silently reads a KV block containing a different token. This was another bug where the model still ran but produced incorrect output.

In the L8 smoke test, a request with a 97-token shared prefix and a small unmatched tail processed 25 query tokens instead of 121 over 20 steps. The cache avoided 96 query tokens, or about 79% of the work in that test, while producing the same output as the uncached run.

Learning with a Coding Agent #

The process was similar to my earlier experience learning about CT scans with Gemini. At the beginning of each lesson, I asked the agent to explain one concept and then kept asking questions until I could restate the data flow myself. I used it to cross-reference the real SGLang and vLLM implementations and to review my code, but I tried to write the implementation before looking at a complete solution.

This distinction mattered. The agent could generate an allocator or a scheduler immediately, but reading that code did not give me the same understanding as debugging why my own cur_len was wrong or why an emoji disappeared from the token stream. The lesson pages preserve the questions, failed implementations and pitfalls for this reason. Those parts are more representative of how I learnt the material than the final source code.

I still do not understand every optimization inside a production serving engine, but I can now follow a request from the HTTP handler to the paged attention kernel and back. That was the gap I wanted to close.

Running It #

The lesson site starts at L0: Architecture overview. To run the code locally:

git clone https://github.com/moomou/mini_sglang
cd mini_sglang
uv sync
uv run python -m scripts.l1_smoke

From there, each lesson has its own smoke test and builds on the same Qwen3 implementation.

since 2017
rss