Hit Record: Qwen3.8 27B's architecture in 3D

It's a dense model, not a mixture of experts: every unit of its neural networks runs for every token. Its release holds 27.8 billion weights in all. The 64 main layers repeat one pattern 16 times: three Gated DeltaNet layers, then one attention layer. Each layer has two steps: the first brings in information from other tokens, and the second, a neural network of 17,408 units, works on each token alone. Inside the layers, each token's working memory is 5,120 numbers, and the model can read up to 262,144 tokens of context.

The release also includes a vision encoder for images and video, and one extra layer for multi-token prediction (MTP), which software that supports it can use to draft tokens ahead for the main model to check. The animation follows text through the 64 main layers and leaves both out.

The record is the model: its weights, pressed into the vinyl. Its groove runs through the 64 layers in order, from the outer edge to the label, and every token takes the whole trip. The screen under the DJ shows the token's working memory, the 5,120 numbers each layer reads and adds to, usually called the residual stream.

Each teal card is one token's keys and values, kept by the attention layers: later tokens match their queries against the keys and read from the values. Together the cards are the key-value cache, or KV cache. The violet notepads are what the Gated DeltaNet layers keep instead, a fixed set of grids each token rewrites. The dots that burst out of the groove are the 17,408 units of a layer's neural network: green adds a unit's learned pattern to the working memory, blue subtracts it. The DJ is a guide, not part of the model, though its visor shows a reading of the working memory, described below.

A model's weights do two jobs: they store what it learned, and they do the work of processing each token. A mixture of experts splits the neural networks in some or all of its layers into many experts and runs only a few of them for each token, so it can store far more than it uses at once. A dense model uses nearly all of its weights on every token.

Qwen3.8 27B uses about 25.6 billion weights for each token it writes. That's about 8.5 times the 3 billion Qwen3.6-35B-A3B uses, and about 1.5 times the 17 billion Qwen reports for Qwen3.7-Plus, a mixture with 397 billion weights. On the benchmark table in Qwen3.8 27B's model card, the 27B scores higher than Qwen3.7-Plus on 10 of its 12 text benchmarks.

Research suggests why the split matters. In Mixture of Parrots, adding experts while keeping the weights used per token fixed kept improving how much a model could memorize, while its reasoning stopped improving; making each layer wider helped reasoning instead. Their tests used synthetic tasks and models much smaller than these, so it's a clue rather than an explanation for any one benchmark.

It doesn't mean dense always wins. Qwen3.8-Flash-Next is a mixture that uses only 6 billion weights per token, and it beats the 27B on most of the tests in its own card. Those models were trained differently, and using fewer weights per token cuts the arithmetic, which is a big reason labs build mixtures.

Gated DeltaNet is a form of linear attention. Instead of keeping something for every earlier token, each of these layers keeps a fixed-size memory: 48 grids of 128 by 128 numbers, the notepads in the animation. That memory is the same size at the 10th token as at the 262,144th, so each new token costs the same work however long the conversation gets. Three of every four layers in Qwen3.8 27B work this way.

Each grid links keys to values, both lists of numbers. For each token, the layer first fades each grid by an amount the token sets. Then it looks up the token's key, compares what comes back with the value the token brings, and writes in the difference, scaled by a strength the token also sets. If the memory already had it right, almost nothing changes. Finally, the token's query reads an answer out of the updated grid.

With no separate entry for each token, the grids hold a compressed summary: fading and later writes weaken or overwrite older information, especially under similar keys.

The other 16 layers use attention: they keep a key and a value for every token so far, so a new token can compare its query with every key and read a weighted mix of the values, from anywhere in the conversation. That cache grows by 32,768 numbers with every token.

The 48 DeltaNet layers' grids hold about 37.7 million numbers in all, as many as the attention layers keep for 1,152 tokens. Past that point the attention cache is the bigger of the two, and it keeps growing while the grids don't. If all 64 layers used attention, the cache would grow four times as fast. The mix keeps every token's keys and values in a quarter of the layers and a compact, fading summary in the rest.

Each attention layer has 24 heads, separate calculations that read the earlier tokens in different ways, and each head's result is 256 numbers. A learned gate multiplies each of those numbers by its own factor between 0 and 1, worked out from the token itself. The heads' results are then combined into 5,120 numbers and added to the working memory. For any one token, the gate can turn part of a head's output, or nearly all of it, almost off.

Each of the 64 layers ends with a neural network of 17,408 units, and every unit runs on every token: there are no experts to choose between. A unit makes two weighted sums of the token's working memory, normalized first. The first sum goes through a sigmoid linear unit, or SiLU, which shrinks negative sums toward zero and leaves large positive ones almost unchanged. That result multiplies the second sum to give the unit's response.

Each unit also has a learned pattern of 5,120 numbers. Its response scales that pattern, which is then added to the working memory; a negative response subtracts it. How strongly each unit responds changes from token to token, but none are skipped, and a unit that adds almost nothing costs as much as one that adds a lot. These networks hold 17.1 billion of the model's 27.8 billion weights, about 62%.

Midway through the layers, a token's working memory is 5,120 numbers, and on their own they don't say what the model is leaning toward. The visor reads them with a Jacobian lens. For each layer, the lens has learned how a change to the numbers there tends to carry through to the last layer, averaged over a lot of ordinary text. It uses that to translate the layer's numbers into the last layer's terms, where the model's own output head can score every token against them. The visor shows the top scores.

The version here is J++, which scored higher than the original lens in its authors' tests and leaves out tokens with no letter or digit, such as punctuation and spaces. It was fitted on Qwen3.8 27B itself, with its authors' code. In my tests on questions with an unstated middle step, like a country you have to work out before you can name its capital, it puts that middle step among its top 10 tokens, at one of the layers checked, on about 61% of the questions it wasn't tuned on.

On this prompt, the middle layers mostly read as the instruction itself, "answer" and "word". "Discs" leads at layer 50, then "records" from layer 59, trading places with "vinyl" in the last few layers. The model's output picks "Records" at 85%, with "Vin", the start of "Vinyl", second. These readings are estimates of what the working memory leans toward, not the model's thoughts word for word, and not why it chose its answer.

Training teaches it to finish a reply with a special token, <|im_end|>, ID 248046. Qwen's generation settings stop as soon as that token is picked, and also on <|endoftext|>, 248044, which ends a document.

I run it on a 16 GB RTX 5070 Ti, in a quantized version (Unsloth's UD-IQ3_S) that stores most of its weights in about 3 bits instead of 16, with its KV cache compressed too. That leaves room for about 120,000 tokens of context. How much memory you need depends on both settings and on the context length; my benchmarks and exact settings are in an earlier post.