Contents 11 sections

A next-token language model writes from left to right. It chooses a token, adds it to the context, then predicts the next one. Each choice has to finish before the next prediction can use it.

Next-concept modeling, or NCM, puts the waiting somewhere else. It first generates a rough, learned representation of the whole output, then adds finer corrections over a fixed number of levels. Every position within a level is predicted together. A decoder turns the result into text.

It is not something to build on yet. The released models have about 100M parameters, every output has a fixed length, and the paper never times NCM against a next-token model. What is worth understanding now is the move itself: the sequential steps move from words to levels of a learned representation. To see why that can work, start with four numbers. Further down, the released TinyStories model shows what its text looks like after every level.

Why the next token waits

A token is a unit of text, often a word or part of one. Suppose the output begins “The cats”. The next word should take account of the plural noun. In a next-token model, it does: each prediction sees the tokens already generated. That feedback is why such models are called autoregressive.

context              choice          next context
"The"            --> "cats"      --> "The cats"
"The cats"       --> "are"       --> "The cats are"
"The cats are"   --> "hungry"    --> "The cats are hungry"

Training can process many positions together because the correct preceding tokens are already in the training example. At generation time, those tokens have not been chosen yet.

Why not choose all the words at once?

Suppose the only sentences in a dataset are “The cat is hungry” and “The cats are hungry”, each appearing equally often. Half the nouns are cat and half are cats. Half the verbs are is and half are are.

Now choose a noun and a verb separately, like flipping two independent coins. That gives four equally likely sentences:

Sentence                 Probability   Grammar
The cat is hungry        25%           correct
The cat are hungry       25%           wrong
The cats is hungry       25%           wrong
The cats are hungry      25%           correct

The two wrong sentences never appeared in the dataset. They arise because the verb choice ignores the noun choice.

If a shared choice first sets the sentence to singular, the noun can be cat and the verb is. If it sets plural, they can be cats and are. The two words can then be chosen together using that shared information. NCM learns shared information as a hierarchy of codes; it does not receive a supplied singular/plural label.

What a code contains

An encoder turns the input tokens into vectors: lists of numbers. A codebook is a learned table of vectors. To quantize a vector, find its nearest entry in that table and store the entry’s integer ID.

Use a single number in place of a vector for a moment:

Codebook

ID:        0       1       2
Value:     0       4       8
                   ^
input 5 -----------+ closest value is 4

Encode: 5 --> code 1
Decode: code 1 --> value 4
Error:  5 - 4 = 1

For real vectors, the lookup minimizes squared distance across all their components. Because the encoder and the codebook are learned together, training decides which distinctions the codes keep. This is the vector quantization of VQ-VAE. NCM uses one codebook per level, each representing what the earlier levels missed.

Represent the error next

Take the target sequence [1, 3, 5, 7]. Its average is 4. A single coarse code with value 4, repeated across the four positions, gives a rough reconstruction:

Target:          [ 1,  3,  5,  7]
Reconstruction:  [ 4,  4,  4,  4]
Residual:        [-3, -1,  1,  3]

The residual is the target minus what has already been represented. At the next level, average the residual in two groups. The left group’s average is -2; the right group’s is 2. If the next codebook contains those values, two codes can add this correction:

Earlier:         [ 4,  4,  4,  4]
Correction:    + [-2, -2,  2,  2]
                 ---------------
Reconstruction:  [ 2,  2,  6,  6]
Residual:        [-1,  1, -1,  1]

One more level, now with a code for every position:

Earlier:         [ 2,  2,  6,  6]
Correction:    + [-1,  1, -1,  1]
                 ---------------
Reconstruction:  [ 1,  3,  5,  7]

The first level captured the overall value, the second a left/right difference, and the third the remaining local differences. These resolutions are the scales: one code position, then two, then four, all describing the same sequence. Quantizing what earlier codes missed is called residual quantization; NCM cites SoundStream for it. The demo below includes this calculation as residual.py.

This used seven codes for four numbers, more than it started with. What it buys is a shared earlier reconstruction that every later level builds on. A generator can use it to coordinate predictions, the way both words used the singular/plural choice.

Learn the representation, then the generator

The example calculated its corrections from the target numbers. During generation there is no target paragraph to subtract from. NCM therefore trains in two stages: first a representation of text in this form, then a generator for its codes.

The first stage trains what the paper calls the tokenizer: an encoder, a quantizer with one codebook per level, and a decoder. It works on top of an ordinary tokenizer that has already split the text into tokens. The encoder maps those tokens to vectors. At each level, the quantizer downsamples the remaining error, picks codes from that level’s codebook, and maps their vectors back to a full-length correction. The decoder learns to recover the original tokens from the sum.

Left alone, training could push most of the information into the late levels. So during tokenizer training NCM sometimes cuts the hierarchy short and makes the decoder reconstruct from the early levels alone. This nested dropout, which the paper takes from ImageFolder, pushes the early levels to carry information the decoder can use.

The second stage freezes the tokenizer and trains a transformer to predict its codes, one level at a time. The input for a level is the reconstruction from the earlier levels, resized to that level’s number of positions. The targets are the codes the tokenizer assigned to the training text. Some conditioning codes are swapped for random codes from the same codebook, so the generator learns to cope with imperfect earlier levels, which it will meet when it generates its own. The paper’s section 3 describes both stages.

NCM adapts this next-scale approach from Visual Autoregressive Modeling, which generates images coarse to fine. An image can be shrunk to a lower resolution. Averaging words does not give a smaller sentence in the same way, so the encoder has to learn a vector space where coarse versions are useful.

What happens during generation

Once trained, the generator builds the hierarchy without any target text. It uses the codes already chosen to predict distributions for every position at the next level, samples those positions, and adds their contributions to the reconstruction.

Earlier reconstruction, at full length
                 |
                 v
Resize to the next level's number of positions
                 |
                 v
Transformer predicts distributions together
          /      |      \
         v       v       v
      [code]  [code]  [code]
          \      |      /
           +-----+-----+
                 |
                 v
Map contributions to full length and add them
                 |
                 v
       Finer reconstruction
                 |
           repeat by level
                 |
                 v
       Final decoder --> text

The levels are sequential. The codes within a level are sampled independently given the earlier reconstruction. The hidden states can attend to other positions in the current input block, but a newly sampled code does not become a neighbour’s input within that same step. So the cat/cats problem is still there inside each level. NCM’s bet is that the earlier levels already carry the shared choices, so codes sampled side by side agree.

“Concept” is the name for a learned code. The training data does not label codes as “character”, “plot”, or “plural noun”. Those interpretations have to be tested against what changing the codes does.

Watching a small model generate

The demo runs the released TinyStories model on CPU and decodes the full-length representation after every level. Those decodes are for inspection only; the generator never sees them. It saves seeds 0, 1 and 42, and the captured output has all three.

Seed 0 is also the sample the fixed-prefix experiment below starts from. These excerpts are its opening at four intermediate levels, cut at the last whole word within 160 characters. Whitespace is collapsed and the lines are wrapped for display. No words have been corrected.

Level 1, 1 code position
. was a. and. the.. .. was... He saw.,. the. the..... . the.... and
saw... The was... ... Tim. the.. help. the.... was. and. the. the..
the..,. ... the the.. .

Level 3, 16 code positions
a time a, a a time, a a a a little girl. Sue saw it, that day. it picked
it it it. He it it up.,. and,.,. to wanted to new. 's said, ",, ", I
want I I want

Level 6, 40 code positions
upon upon a time, in a, a, a a named girl. Lucy saw a, purple ball. She
picked it it it up up up said said " ". the remember to!"!" The box..
Spot Amy's mom mom

Level 9, 76 code positions
Once upon a time, in a small house, a little girl named Lucy saw a big,
purple purple box. She picked it up and looked up up. "Wow, the is is!"
a a named Spot.

Across the whole text, some words settle early: “box” is in the decoded text from level 2 and “mail” from level 4. Others change later. The girl is Sue at level 3 and Lucy from level 4, and the object in the first sentence is a ball at levels 5 and 6 and a box from level 7. The coarse decoded text is not a readable outline. It contains repeated fragments, and later levels change some of the apparent choices.

The full final output, with line wrapping added:

Once upon a time, in a small house, a little girl named Lucy saw a big,
purple purple box. She picked it up and looked at it. "Wow, this box
is!" she called called Spot.

One day, Lucy's mom said, "Lucy, I am going to mail this mail to my
friend Spot." Spot wagged his tail and barked. Lucy knew what Spot could
help. They started to put the mail in the mail, and carried the box.

On the mail letters, Lucy's mom came into the house and saw the box. She
opened the box and said, "Thank you, Spot

Repetition and broken phrasing remain. The sample fills the fixed 128-token output and stops inside a quotation. The doubled words, “purple purple” and “called called”, look like the cat/cats problem from the start: neighbouring positions settled on the same word without seeing each other’s choice. That is my guess. This run cannot separate parallel sampling within a level from the decoder reading two nearly equal vectors.

What survives when the coarse codes stay fixed

The paper’s Appendix H fixes the coarse codes of supplied stories and generates new fine levels. Its examples tend to preserve broad themes while changing details. The demo tries a variation: fix the early codes of a generated sample and see what changes.

The first three levels contain 25 codes. Keeping those exact codes from seed 0, then generating the remaining nine levels with two different seeds, gives these opening paragraphs:

Seed 1
Once upon a time, in a small house, a girl named Lily went in the
kitchen. Lily saw her toy phone. She picked it up and went to study. She
was very impatient to find something new to eat.

Seed 2
Once upon a time, in a modest house, lived a girl named Sue. One day,
Sue found purse on the floor. She picked it up and showed it to her
friends. "Look Tom! I found a purse!"

The original has Lucy, a purple box and a mail scene. Seed 1 has Lily, a toy phone and a toy box; seed 2 has Sue, a purse and her friend Tom. All three open with a girl in a house who picks up an object. The mail scene is gone from both variants, and Spot appears in only one. One later beat recurs at the same point in all three, about four fifths of the way through: the girl’s mom comes into the house, the room or the park.

The script checks that the early codes are identical. The resemblance could also reflect common TinyStories patterns, and one prefix with two continuations cannot separate the two or show what any single code means.

How much work is one level?

The released TinyStories model writes 128 tokens in twelve levels:

level:       1   2   3   4   5   6   7   8   9  10  11  12
positions:   1   8  16  24  32  40  52  64  76  92 108 128

The first level’s single code is drawn from a fixed distribution stored in the checkpoint, level0_unigram, where the paper’s general algorithm predicts it. The transformer generates the other eleven levels, and one decoder pass produces the tokens: twelve sequential steps instead of 128. The steps are not equal. The last predicts 128 positions at once, and the levels hold 641 code positions in all, more than the text has tokens, so the step count alone does not give a speedup. The demo’s CPU timings are not a benchmark.

What the results show

The authors report generation 21x faster than DUO on LM1B and 31x faster than AURORA-LM on XSum at comparable or better quality, and 10x faster on OpenWebText at quality they call competitive. All three baselines are diffusion models, timed on an H100 at batch size 16. Abstract, section 4.6.

Next-token models are compared on quality only, and the result is mixed. On TinyStories, a next-token baseline’s generative perplexity, how predictable GPT-2 large finds the samples, is 8.89 against NCM’s 15.41. Lower is better, though repetition also lowers it. The baseline takes 128 forward passes to NCM’s 12. Next-token baselines also lead on OpenWebText and WMT14 translation; NCM leads on XSum summarization. Section 4.2, Table 2, Table 8.

Most generators in the paper have about 100M parameters; the largest has 725M. The hierarchy’s depth and lengths are fixed. The tokenizer is trained separately and gets some tokens wrong, 0.02% on TinyStories and 1.21% on WMT14; one wrong name or digit can change the meaning. The authors also report occasional artifacts at coarse-level transitions. Table 1, Appendix I.

Try changing only the fine levels

Clone the demo and run it using its README. Then change keep_levels=3 to keep_levels=4 in the variant-generation call in demo.py and save the output to a new file. That holds 49 codes fixed instead of 25.

Compare the two sets of continuations. Look for details that survive, details that change, and places where the story stops making sense. Inspect the difference before deciding what you believe a “concept” has captured.

References