Skip to content

THE FIELD JOURNAL

How Do LLMs Work? A Plain-English Guide to Tokens, Weights and Why AI Makes Things Up

By Local AI Scout · Beginner guide

Hand-drawn teacher's desk with a laptop chat window sending a row of colored token tiles into a small chip, illustrating how LLMs work

How do LLMs work? They predict the next token, one at a time

A large language model (LLM) works by predicting the next small chunk of text, called a token, from everything that came before it. It adds that token to the text and predicts again, one token at a time, until the reply is done. The predictions come from billions of numbers, called weights, tuned during training on huge amounts of text.

Google's course calls LLMs essentially autocomplete mechanisms, and that holds for ChatGPT and for the free models you can run on a laptop. What differs is size, training and whose computer does the work.

This guide follows one reader: a middle school science teacher who uses ChatGPT for quiz questions and parent emails. She has a laptop with 16GB of memory, keeps hearing about "tokens" and "parameters", and wants to know whether a free model on it is good enough for her work.

The jargon, translated:

TermPlain English
LLM (large language model)A program trained on vast amounts of text to predict what comes next
TokenA chunk of text: a word, part of a word, or a symbol
Parameters, weightsThe billions of learned numbers that make up the model file
TrainingThe costly, one-off process that sets those numbers
InferenceUsing the finished model; the part your laptop does
Context windowHow much text the model can see at once
HallucinationA fluent, confident answer that is false
Fine-tuningExtra training on narrower examples
RLHFFine-tuning guided by people ranking answers
QuantizationStoring each number with fewer bits to shrink the file
Open-weightWeights published for anyone to download and run
HarnessThe app that runs the model, like Ollama

What is an LLM? A huge file of numbers, and an app to run it

An LLM is a neural network trained on text. Today's LLMs use a design called the Transformer, from the 2017 paper "Attention Is All You Need", which proposed an architecture based solely on attention mechanisms. Attention works out which words matter to each other. Google's course says self-attention asks, for every token, how much each other token affects its interpretation.

What the model learned is stored in its parameters, the numbers adjusted during training. Weights are the main kind, and people use the two words interchangeably. Picture a mixing desk with billions of tiny dials: training turns each one until the output sounds right, and the finished settings are the file you download. Google's course says Transformers contain hundreds of billions or even trillions of parameters, though the models people run at home are far smaller.

The file does nothing alone. An app loads it, feeds it your text and shows the reply. That app is an AI harness, such as Ollama or LM Studio. ChatGPT is both pieces in one, running on OpenAI's computers.

What "open-weight" means

Open-weight means the trained weights are published for anyone to download and run. That is not fully open source. The Open Source Initiative describes open weights as the final weights and biases of a trained neural network, which come without the training code and the training dataset. OpenAI calls gpt-oss its first open-weight language models since GPT-2, so the models behind ChatGPT itself stay on OpenAI's servers. Licenses vary by family: gpt-oss uses the Apache 2.0 license, and so does Gemma 4.

What are tokens, and how is the next one chosen?

How a sentence splits into tokens

A token is the unit an LLM reads and writes: often a whole word, sometimes a piece of one or a punctuation mark. Each token has an ID number, and the model only works with the numbers.

Hand-drawn diagram of the sentence Using a Transformer network is simple split into seven tokens, with transform and ##er as two pieces, each mapped to an ID number
A tokenizer cuts text into tokens and swaps each one for an ID number; the model only ever sees the numbers.

In Hugging Face's course, the BERT model's tokenizer splits "Using a Transformer network is simple" into seven tokens: Using, a, transform, ##er, network, is, simple, where ## marks a piece that continues the previous word. Those become the IDs 7993, 170, 11303, 1200, 2443, 1110 and 3014. Common words stay whole because frequently used words should not be split. Each model family has its own tokenizer, and the one for gpt-oss has 201,088 tokens in its vocabulary.

OpenAI's rule of thumb is that one token is about four characters of English, and 100 tokens is about 75 words. Splitting is language-dependent, and other languages often need more tokens for the same text.

How the model picks the next token

Say the teacher types "Please remember to hand in your". The model scores every token in its vocabulary, and the scores become probabilities. Made-up numbers, to show the idea:

  • homework: 48%
  • permission (as in permission slip): 15%
  • essays: 9%
  • lunch: under 1%

The app picks one, say "homework", adds it, and predicts again to get the next token, perhaps "by". That loop is the whole trick. Apps don't always take the top choice; a temperature setting, which Hugging Face's course calls a creativity dial, makes the pick more or less random. That is why, in Google's glossary wording, LLMs are usually nondeterministic: the same question often gets different wording.

No step in that loop looks anything up: "homework" wins because it is likely, not because anything was checked.

What happens when you press Enter

Here is the full trip for "Write five quiz questions on photosynthesis", the same in ChatGPT or on her laptop.

Hand-drawn loop showing what happens when you press Enter in ChatGPT: tokenizer, model, scores, pick one token, add it and repeat until an end token
Every reply is one loop run over and over: score the next token, pick one, add it, repeat.
  1. The app assembles the prompt: any instructions of its own, the earlier conversation and the new message, in the model's chat template, which marks who said what. OpenAI's docs say each text generation request is independent and stateless, so the whole conversation is fed to the model again every time.
  2. The tokenizer turns that text into token IDs.
  3. The model reads all the tokens at once, the prefill phase, and its attention layers work out how they relate.
  4. It scores every possible next token, and the app picks one, nudged by temperature.
  5. The token is added and the model goes again, generating one token at a time. A KV cache stores and reuses earlier calculations, saving time but taking memory.
  6. It stops at a special end-of-sequence token or a length limit.
  7. The app turns each token back into text as it arrives, which is why replies appear word by word.
  8. The model itself does not change: prompt engineering doesn't alter the model's parameters, as Google's course notes.

What a context window is, and why long chats lose track

The context window is the number of tokens a model can process in a given prompt, in Google's glossary wording, and OpenAI's docs say the limit covers input, output and reasoning tokens. Anything outside it is invisible to the model.

gpt-oss-20b supports context lengths of up to 128k, and Gemma 4's 12B, 26B and 31B list 256K tokens. Your app may use far less: Ollama defaults to 4k tokens on machines with under 24 GiB of VRAM, because a longer context needs more memory. That is about 3,000 words, input and reply together. For the teacher, a 20-page unit plan won't fit at the default. She can raise it with the slider in Ollama's settings if memory allows, or work in sections.

Training vs inference, fine-tuning and RLHF

Training sets the weights. Inference is using the finished model, and it is all your laptop does when you chat.

Pretraining comes first. Hugging Face's course explains that the weights are randomly initialized, then the model learns by predicting the next word across very large amounts of text. Scale makes it expensive: OpenAI's model card says gpt-oss was trained on a text-only dataset with trillions of tokens, and the 120b run required 2.1 million H100-hours, with the 20b needing almost 10 times fewer. Training also fixes a date: gpt-oss has a knowledge cutoff of June 2024.

A freshly pretrained base model continues text but doesn't follow instructions well. Fine-tuning, which Google's course describes as training on examples specific to the task, turns it into an assistant using examples of helpful conversations.

RLHF, reinforcement learning from human feedback, adds people. Hugging Face's explainer lists three core steps: pretrain a language model, train a reward model from human rankings of its answers, then fine-tune the model with reinforcement learning. In OpenAI's 2022 InstructGPT paper, outputs from the 1.3B parameter InstructGPT model are preferred to those of the 175B GPT-3. gpt-oss went through a supervised fine-tuning stage and a high-compute RL stage.

This stage is also where a model learns what to refuse. Our uncensored AI guide explains how alignment training shapes refusals. The model the teacher downloads has been through all of this; she runs it, not trains it.

Why bigger models need more memory, and what quantization does

Every parameter is a number held in memory while the model runs, so more parameters means more memory. Hugging Face's rule of thumb is roughly 2 GB of VRAM for every billion parameters at 16-bit precision. VRAM is the fast memory on a graphics card.

Bar chart of Gemma 4 31B memory needs: 69.9 GB at 16-bit, 34.9 GB at 8-bit and 17.5 GB at 4-bit, showing how quantization shrinks an LLM

Quantization shrinks that by storing each weight with fewer bits, a bit like rounding 0.73914 to 0.74. Hugging Face describes it as storing the weights in a lower precision while preserving as much accuracy as possible. Google's figures for Gemma 4 31B show the effect: 69.9 GB at 16-bit, 34.9 GB at 8-bit and 17.5 GB at 4-bit. llama.cpp's docs list Llama 3.1 8B at 14.96 GiB in F16 and 4.58 GiB in Q4_K_M, a popular 4-bit level, and warn that the process may introduce some accuracy loss. To read labels like Q4_K_M and pick a file, see GGUF quantization explained.

Two catches. gpt-oss-20b activates 3.6B parameters per token out of 21B, which makes it quick, but for a mixture-of-experts model like Gemma 4 26B, Google says all 26 billion parameters must be loaded into memory. Plan for the total. And Google's estimates don't include the additional VRAM needed for supporting software or the context window.

Sizes at about 4 bits for a few popular open-weight models. The sources measure differently, so treat them as ballpark; the last column is my rough guide.

ModelParametersSize at about 4-bitMemory to plan for
Gemma 4 E4B4.5B effective (8B with embeddings)4.5 GB (Q4_0, Google estimate)8GB is tight; 16GB is comfortable
Llama 3.1 8B8B4.58 GiB (Q4_K_M, llama.cpp docs)8GB
Gemma 4 12B11.95B6.7 GB (Q4_0, Google estimate)16GB
gpt-oss-20b21B, 3.6B active14 GB download (MoE weights in MXFP4, 4.25 bits per parameter)16GB, tight
Qwen3.8-27B27B18 GB (q4_K_M, Ollama download)24GB or more
Gemma 4 31B30.7B17.5 GB (Q4_0, Google estimate)24GB or more
gpt-oss-120b117B, 5.1B active65 GB downloadA single 80GB GPU

For the teacher's 16GB laptop, Gemma 4 12B or gpt-oss-20b is the ceiling. Google calls the 12B small enough to run locally with just 16GB of VRAM or unified memory, and OpenAI says gpt-oss-20b only requires 16GB, so expect a tight fit. Our guide to running a ChatGPT-style model locally covers the gpt-oss-20b setup.

Why LLMs make things up, and how to check their answers

LLMs make things up because they generate plausible text, not checked facts. OpenAI defines hallucinations as plausible but false statements generated by language models, and they arrive in the same confident tone as true ones.

OpenAI's researchers give two reasons. Arbitrary low-frequency facts, like a pet's birthday, can't be learned from patterns alone. And standard training and evaluation procedures reward guessing over acknowledging uncertainty. The teacher knows the incentive: on a test with no credit for a blank, guessing pays.

Smaller models know less. OpenAI's model card says smaller models have less world knowledge than larger frontier models and tend to hallucinate more. On its PersonQA test of facts about people, with browsing off, gpt-oss-20b answered 15.5% correctly with a 53.2% hallucination rate, against 35.6% and 36.1% for o4-mini.

Habits that catch most problems:

These habits significantly reduce hallucinations but don't eliminate them, Anthropic adds. For the teacher, every answer key gets checked against the textbook. A parent email is lower risk, because she already knows the facts.

Local LLM on your laptop vs ChatGPT in the cloud

The mechanism is identical. What changes is whose computer runs it and how big the model is.

Hand-drawn comparison of a local LLM on a laptop inside a house with a padlock and ChatGPT on cloud servers, both running the same token loop
The same next-token loop runs in both places; what changes is whose computer runs it and how big the model is.

Privacy

Local wins. Ollama's FAQ says it doesn't see your prompts or data when you run locally. On ChatGPT Free, Plus and Pro, OpenAI may use your content to train its models unless you turn off "Improve the model for everyone" in Data controls; Business, Enterprise, Edu and API data are not used by default. Local still goes online for downloads, and Ollama's optional cloud models send prompts to its servers; setting OLLAMA_NO_CLOUD=1 turns off Ollama's cloud features. Student names and grades belong on the laptop.

Speed

Speed depends on your hardware. A model that fits in fast GPU memory runs well; one that spills into ordinary memory slows down, which is why Ollama's docs advise you to avoid offloading the model to CPU. Our local vs cloud AI comparison has tokens-per-second figures by hardware.

Quality

ChatGPT wins on hard problems. gpt-oss-20b scores 85.3 on the MMLU benchmark against 93.0 for OpenAI's o4-mini, a model ChatGPT has since retired in favor of newer ones. Then there is the hallucination gap, a fixed knowledge cutoff, the short default context, and no web access unless you set it up. For the teacher, drafting, rewording and summarizing her own notes are well within reach of a 12B or 20B model; obscure facts are where she should expect misses.

Common questions about how LLMs work

How do LLMs work in simple terms?

An LLM reads your text as tokens and predicts the most likely next token using billions of numbers learned in training. It adds that token and predicts again until the reply ends. It does not look facts up; it generates text that fits learned patterns.

What are tokens in an LLM?

Tokens are the chunks of text a model reads and writes: whole words, word pieces, spaces or punctuation, each mapped to an ID number. OpenAI's rule of thumb is that one token is about four characters of English, so 100 tokens is roughly 75 words.

Why do LLMs hallucinate?

An LLM generates the most plausible next words, not verified facts, so it can state false things confidently. OpenAI's researchers say rare facts can't be learned from patterns alone, and that training rewards guessing over admitting uncertainty. Smaller models, like many local ones, know less and tend to hallucinate more.

How much memory do I need to run an LLM locally?

At 4-bit quantization, plan on a little over half a gigabyte per billion parameters, plus room for context and other apps. An 8GB machine suits models up to about 8B, 16GB suits Gemma 4 12B or gpt-oss-20b, and 30B-class models want 24GB or more.

Is a local LLM as good as ChatGPT?

For drafting, rewording and summarizing your own text, a good 12B to 20B local model is often close enough. For hard reasoning, long documents and obscure facts, ChatGPT does better: OpenAI's model card scores gpt-oss-20b at 85.3 on MMLU against 93.0 for o4-mini, an older ChatGPT model. Local wins on privacy.

Verdict: learn the loop, then test a small model on your own work

An LLM is a next-token predictor built from billions of trained weights. Tokens set its limits, memory sets the model size, and hallucinations come with the method.

For the teacher, a local model is good enough for the writing half of her job: parent emails, reworded worksheets, first drafts of quiz questions she will check. I would keep ChatGPT for research-style questions and check anything factual from either one. Fine-tuning and new hardware can wait.

One next step: install Ollama, run ollama run gemma4:12b (about an 8GB download), and give it three tasks she did in ChatGPT last week. Comparing the answers side by side will tell her more than any benchmark.

Sources

Keep exploring: Apps · Models · All guides

THE SCOUT LETTER

Good finds.
Less noise.

Useful local AI apps, practical guides and the updates worth your time. A thoughtful note in your inbox.

Free to join. Unsubscribe with one click.