How do LLMs work? They predict the next token, one at a time
A large language model (LLM) works by predicting the next small chunk of text, called a token, from everything that came before it. It adds that token to the text and predicts again, one token at a time, until the reply is done. The predictions come from billions of numbers, called weights, tuned during training on huge amounts of text.
Google's course calls LLMs essentially autocomplete mechanisms, and that holds for ChatGPT and for the free models you can run on a laptop. What differs is size, training and whose computer does the work.
This guide follows one reader: a middle school science teacher who uses ChatGPT for quiz questions and parent emails. She has a laptop with 16GB of memory, keeps hearing about "tokens" and "parameters", and wants to know whether a free model on it is good enough for her work.
The jargon, translated:
| Term | Plain English |
|---|---|
| LLM (large language model) | A program trained on vast amounts of text to predict what comes next |
| Token | A chunk of text: a word, part of a word, or a symbol |
| Parameters, weights | The billions of learned numbers that make up the model file |
| Training | The costly, one-off process that sets those numbers |
| Inference | Using the finished model; the part your laptop does |
| Context window | How much text the model can see at once |
| Hallucination | A fluent, confident answer that is false |
| Fine-tuning | Extra training on narrower examples |
| RLHF | Fine-tuning guided by people ranking answers |
| Quantization | Storing each number with fewer bits to shrink the file |
| Open-weight | Weights published for anyone to download and run |
| Harness | The app that runs the model, like Ollama |
What is an LLM? A huge file of numbers, and an app to run it
An LLM is a neural network trained on text. Today's LLMs use a design called the Transformer, from the 2017 paper "Attention Is All You Need", which proposed an architecture based solely on attention mechanisms. Attention works out which words matter to each other. Google's course says self-attention asks, for every token, how much each other token affects its interpretation.
What the model learned is stored in its parameters, the numbers adjusted during training. Weights are the main kind, and people use the two words interchangeably. Picture a mixing desk with billions of tiny dials: training turns each one until the output sounds right, and the finished settings are the file you download. Google's course says Transformers contain hundreds of billions or even trillions of parameters, though the models people run at home are far smaller.
The file does nothing alone. An app loads it, feeds it your text and shows the reply. That app is an AI harness, such as Ollama or LM Studio. ChatGPT is both pieces in one, running on OpenAI's computers.
What "open-weight" means
Open-weight means the trained weights are published for anyone to download and run. That is not fully open source. The Open Source Initiative describes open weights as the final weights and biases of a trained neural network, which come without the training code and the training dataset. OpenAI calls gpt-oss its first open-weight language models since GPT-2, so the models behind ChatGPT itself stay on OpenAI's servers. Licenses vary by family: gpt-oss uses the Apache 2.0 license, and so does Gemma 4.
What are tokens, and how is the next one chosen?
How a sentence splits into tokens
A token is the unit an LLM reads and writes: often a whole word, sometimes a piece of one or a punctuation mark. Each token has an ID number, and the model only works with the numbers.

In Hugging Face's course, the BERT model's tokenizer splits "Using a Transformer network is simple" into seven tokens: Using, a, transform, ##er, network, is, simple, where ## marks a piece that continues the previous word. Those become the IDs 7993, 170, 11303, 1200, 2443, 1110 and 3014. Common words stay whole because frequently used words should not be split. Each model family has its own tokenizer, and the one for gpt-oss has 201,088 tokens in its vocabulary.
OpenAI's rule of thumb is that one token is about four characters of English, and 100 tokens is about 75 words. Splitting is language-dependent, and other languages often need more tokens for the same text.
How the model picks the next token
Say the teacher types "Please remember to hand in your". The model scores every token in its vocabulary, and the scores become probabilities. Made-up numbers, to show the idea:
- homework: 48%
- permission (as in permission slip): 15%
- essays: 9%
- lunch: under 1%
The app picks one, say "homework", adds it, and predicts again to get the next token, perhaps "by". That loop is the whole trick. Apps don't always take the top choice; a temperature setting, which Hugging Face's course calls a creativity dial, makes the pick more or less random. That is why, in Google's glossary wording, LLMs are usually nondeterministic: the same question often gets different wording.
No step in that loop looks anything up: "homework" wins because it is likely, not because anything was checked.
What happens when you press Enter
Here is the full trip for "Write five quiz questions on photosynthesis", the same in ChatGPT or on her laptop.

- The app assembles the prompt: any instructions of its own, the earlier conversation and the new message, in the model's chat template, which marks who said what. OpenAI's docs say each text generation request is independent and stateless, so the whole conversation is fed to the model again every time.
- The tokenizer turns that text into token IDs.
- The model reads all the tokens at once, the prefill phase, and its attention layers work out how they relate.
- It scores every possible next token, and the app picks one, nudged by temperature.
- The token is added and the model goes again, generating one token at a time. A KV cache stores and reuses earlier calculations, saving time but taking memory.
- It stops at a special end-of-sequence token or a length limit.
- The app turns each token back into text as it arrives, which is why replies appear word by word.
- The model itself does not change: prompt engineering doesn't alter the model's parameters, as Google's course notes.
What a context window is, and why long chats lose track
The context window is the number of tokens a model can process in a given prompt, in Google's glossary wording, and OpenAI's docs say the limit covers input, output and reasoning tokens. Anything outside it is invisible to the model.
gpt-oss-20b supports context lengths of up to 128k, and Gemma 4's 12B, 26B and 31B list 256K tokens. Your app may use far less: Ollama defaults to 4k tokens on machines with under 24 GiB of VRAM, because a longer context needs more memory. That is about 3,000 words, input and reply together. For the teacher, a 20-page unit plan won't fit at the default. She can raise it with the slider in Ollama's settings if memory allows, or work in sections.
Training vs inference, fine-tuning and RLHF
Training sets the weights. Inference is using the finished model, and it is all your laptop does when you chat.
Pretraining comes first. Hugging Face's course explains that the weights are randomly initialized, then the model learns by predicting the next word across very large amounts of text. Scale makes it expensive: OpenAI's model card says gpt-oss was trained on a text-only dataset with trillions of tokens, and the 120b run required 2.1 million H100-hours, with the 20b needing almost 10 times fewer. Training also fixes a date: gpt-oss has a knowledge cutoff of June 2024.
A freshly pretrained base model continues text but doesn't follow instructions well. Fine-tuning, which Google's course describes as training on examples specific to the task, turns it into an assistant using examples of helpful conversations.
RLHF, reinforcement learning from human feedback, adds people. Hugging Face's explainer lists three core steps: pretrain a language model, train a reward model from human rankings of its answers, then fine-tune the model with reinforcement learning. In OpenAI's 2022 InstructGPT paper, outputs from the 1.3B parameter InstructGPT model are preferred to those of the 175B GPT-3. gpt-oss went through a supervised fine-tuning stage and a high-compute RL stage.
This stage is also where a model learns what to refuse. Our uncensored AI guide explains how alignment training shapes refusals. The model the teacher downloads has been through all of this; she runs it, not trains it.
Why bigger models need more memory, and what quantization does
Every parameter is a number held in memory while the model runs, so more parameters means more memory. Hugging Face's rule of thumb is roughly 2 GB of VRAM for every billion parameters at 16-bit precision. VRAM is the fast memory on a graphics card.

Quantization shrinks that by storing each weight with fewer bits, a bit like rounding 0.73914 to 0.74. Hugging Face describes it as storing the weights in a lower precision while preserving as much accuracy as possible. Google's figures for Gemma 4 31B show the effect: 69.9 GB at 16-bit, 34.9 GB at 8-bit and 17.5 GB at 4-bit. llama.cpp's docs list Llama 3.1 8B at 14.96 GiB in F16 and 4.58 GiB in Q4_K_M, a popular 4-bit level, and warn that the process may introduce some accuracy loss. To read labels like Q4_K_M and pick a file, see GGUF quantization explained.
Two catches. gpt-oss-20b activates 3.6B parameters per token out of 21B, which makes it quick, but for a mixture-of-experts model like Gemma 4 26B, Google says all 26 billion parameters must be loaded into memory. Plan for the total. And Google's estimates don't include the additional VRAM needed for supporting software or the context window.
Sizes at about 4 bits for a few popular open-weight models. The sources measure differently, so treat them as ballpark; the last column is my rough guide.
| Model | Parameters | Size at about 4-bit | Memory to plan for |
|---|---|---|---|
| Gemma 4 E4B | 4.5B effective (8B with embeddings) | 4.5 GB (Q4_0, Google estimate) | 8GB is tight; 16GB is comfortable |
| Llama 3.1 8B | 8B | 4.58 GiB (Q4_K_M, llama.cpp docs) | 8GB |
| Gemma 4 12B | 11.95B | 6.7 GB (Q4_0, Google estimate) | 16GB |
| gpt-oss-20b | 21B, 3.6B active | 14 GB download (MoE weights in MXFP4, 4.25 bits per parameter) | 16GB, tight |
| Qwen3.8-27B | 27B | 18 GB (q4_K_M, Ollama download) | 24GB or more |
| Gemma 4 31B | 30.7B | 17.5 GB (Q4_0, Google estimate) | 24GB or more |
| gpt-oss-120b | 117B, 5.1B active | 65 GB download | A single 80GB GPU |
For the teacher's 16GB laptop, Gemma 4 12B or gpt-oss-20b is the ceiling. Google calls the 12B small enough to run locally with just 16GB of VRAM or unified memory, and OpenAI says gpt-oss-20b only requires 16GB, so expect a tight fit. Our guide to running a ChatGPT-style model locally covers the gpt-oss-20b setup.
Why LLMs make things up, and how to check their answers
LLMs make things up because they generate plausible text, not checked facts. OpenAI defines hallucinations as plausible but false statements generated by language models, and they arrive in the same confident tone as true ones.
OpenAI's researchers give two reasons. Arbitrary low-frequency facts, like a pet's birthday, can't be learned from patterns alone. And standard training and evaluation procedures reward guessing over acknowledging uncertainty. The teacher knows the incentive: on a test with no credit for a blank, guessing pays.
Smaller models know less. OpenAI's model card says smaller models have less world knowledge than larger frontier models and tend to hallucinate more. On its PersonQA test of facts about people, with browsing off, gpt-oss-20b answered 15.5% correctly with a 53.2% hallucination rate, against 35.6% and 36.1% for o4-mini.
Habits that catch most problems:
- Check names, dates, numbers, quotes and any links it gives against a primary source.
- Paste in the source text and tell the model to answer only from it. Anthropic's docs suggest restricting the model to information from provided documents, and for long documents, asking for word-for-word quotes first.
- Say it may answer "I don't know". Anthropic says explicitly giving permission to admit uncertainty can drastically reduce false information.
- Ask twice. Anthropic's docs treat inconsistent outputs from the same prompt as a warning sign.
These habits significantly reduce hallucinations but don't eliminate them, Anthropic adds. For the teacher, every answer key gets checked against the textbook. A parent email is lower risk, because she already knows the facts.
Local LLM on your laptop vs ChatGPT in the cloud
The mechanism is identical. What changes is whose computer runs it and how big the model is.

Privacy
Local wins. Ollama's FAQ says it doesn't see your prompts or data when you run locally. On ChatGPT Free, Plus and Pro, OpenAI may use your content to train its models unless you turn off "Improve the model for everyone" in Data controls; Business, Enterprise, Edu and API data are not used by default. Local still goes online for downloads, and Ollama's optional cloud models send prompts to its servers; setting OLLAMA_NO_CLOUD=1 turns off Ollama's cloud features. Student names and grades belong on the laptop.
Speed
Speed depends on your hardware. A model that fits in fast GPU memory runs well; one that spills into ordinary memory slows down, which is why Ollama's docs advise you to avoid offloading the model to CPU. Our local vs cloud AI comparison has tokens-per-second figures by hardware.
Quality
ChatGPT wins on hard problems. gpt-oss-20b scores 85.3 on the MMLU benchmark against 93.0 for OpenAI's o4-mini, a model ChatGPT has since retired in favor of newer ones. Then there is the hallucination gap, a fixed knowledge cutoff, the short default context, and no web access unless you set it up. For the teacher, drafting, rewording and summarizing her own notes are well within reach of a 12B or 20B model; obscure facts are where she should expect misses.
Common questions about how LLMs work
How do LLMs work in simple terms?
An LLM reads your text as tokens and predicts the most likely next token using billions of numbers learned in training. It adds that token and predicts again until the reply ends. It does not look facts up; it generates text that fits learned patterns.
What are tokens in an LLM?
Tokens are the chunks of text a model reads and writes: whole words, word pieces, spaces or punctuation, each mapped to an ID number. OpenAI's rule of thumb is that one token is about four characters of English, so 100 tokens is roughly 75 words.
Why do LLMs hallucinate?
An LLM generates the most plausible next words, not verified facts, so it can state false things confidently. OpenAI's researchers say rare facts can't be learned from patterns alone, and that training rewards guessing over admitting uncertainty. Smaller models, like many local ones, know less and tend to hallucinate more.
How much memory do I need to run an LLM locally?
At 4-bit quantization, plan on a little over half a gigabyte per billion parameters, plus room for context and other apps. An 8GB machine suits models up to about 8B, 16GB suits Gemma 4 12B or gpt-oss-20b, and 30B-class models want 24GB or more.
Is a local LLM as good as ChatGPT?
For drafting, rewording and summarizing your own text, a good 12B to 20B local model is often close enough. For hard reasoning, long documents and obscure facts, ChatGPT does better: OpenAI's model card scores gpt-oss-20b at 85.3 on MMLU against 93.0 for o4-mini, an older ChatGPT model. Local wins on privacy.
Verdict: learn the loop, then test a small model on your own work
An LLM is a next-token predictor built from billions of trained weights. Tokens set its limits, memory sets the model size, and hallucinations come with the method.
For the teacher, a local model is good enough for the writing half of her job: parent emails, reworded worksheets, first drafts of quiz questions she will check. I would keep ChatGPT for research-style questions and check anything factual from either one. Fine-tuning and new hardware can wait.
One next step: install Ollama, run ollama run gemma4:12b (about an 8GB download), and give it three tasks she did in ChatGPT last week. Comparing the answers side by side will tell her more than any benchmark.
Sources
- LLMs: What's a large language model? - Google for Developers
- Attention Is All You Need (Vaswani et al., 2017) - arXiv
- Open Weights: not quite what you've been told - Open Source Initiative
- Introducing gpt-oss - OpenAI
- google/gemma-4-E4B-it model card - Hugging Face / Google
- Tokenizers, LLM Course chapter 2 - Hugging Face
- gpt-oss-120b and gpt-oss-20b Model Card - OpenAI
- What are tokens and how to count them? - OpenAI Help Center
- Deep dive into Text Generation Inference with LLMs - Hugging Face
- Machine Learning Glossary - Google for Developers
- Chat templates - Hugging Face Transformers
- Conversation state - OpenAI
- LLMs: Fine-tuning, distillation, and prompt engineering - Google for Developers
- Context length - Ollama
- How do Transformers work? - Hugging Face
- Illustrating Reinforcement Learning from Human Feedback (RLHF) - Hugging Face
- Training language models to follow instructions with human feedback (Ouyang et al., 2022) - arXiv
- Optimizing LLMs for Speed and Memory - Hugging Face Transformers
- Quantization overview - Hugging Face Transformers
- Gemma 4 model overview - Google
- llama.cpp quantize README - ggml-org/llama.cpp
- openai/gpt-oss-20b model card - Hugging Face / OpenAI
- gpt-oss - Ollama
- Qwen/Qwen3.8-27B model card - Hugging Face / Qwen
- qwen3.8 tags - Ollama
- Introducing Gemma 4 12B - Google
- Why language models hallucinate - OpenAI
- Reduce hallucinations - Anthropic
- FAQ - Ollama
- How your data is used to improve model performance - OpenAI Help Center
- Retiring GPT-4o and other ChatGPT models - OpenAI Help Center
- Download Ollama - Ollama
- gemma4 - Ollama
