Skip to content

THE FIELD JOURNAL

Ollama vs llama.cpp vs vLLM: Which Local AI Engine to Use, and When

By Local AI Scout · Beginner guide

Hand-drawn illustration of a terminal window and an app window feeding one laptop chip, for a comparison of Ollama vs llama.cpp vs vLLM

Ollama vs llama.cpp vs vLLM: which should you use?

Use Ollama if you want the easiest way to run models for yourself; it now wraps llama.cpp's server program. Use llama.cpp directly for the newest features and full control of GGUF files. Use vLLM, a Linux serving engine, only when many people or apps send requests at once, because its throughput scales with concurrent users.

This guide follows one reader: a gamer with a Windows 11 PC and either an RTX 4070 with 12GB of VRAM (the fast memory on the graphics card) or a used RTX 3090 with 24GB. They run models in Ollama today and wonder whether to switch engines, and whether vLLM is worth it to serve the family or a small team from a home server.

Choose Ollama if:

Choose llama.cpp if:

Choose vLLM if:

How Ollama, llama.cpp and vLLM compare in October 2026

All three are inference engines: programs that load a model's weights, generate text and offer it through an API, a web address other software can call. In October 2026 the current releases are Ollama v0.40.13, released October 28, llama.cpp v0.7.0, released October 15, with new numbered builds several times a day, and vLLM v0.32.0, released October 18.

FeatureOllamallama.cppvLLM
LicenseMITMITApache 2.0
Install on Windows 11Easy: installer or one PowerShell lineEasy: install script, winget or the Llama appHarder: WSL or Docker Desktop with WSL 2
InterfaceMac and Windows app, plus command lineBuilt-in web UI, command line, Llama appCommand line and API; add Open WebUI for chat
HardwareNVIDIA (compute 5.0+), AMD via ROCm or Vulkan, Apple Metal, CPUCUDA, HIP for AMD, Vulkan, SYCL for Intel, Metal, CPUNVIDIA (7.5+), AMD RX 7900 and RX 9000, Intel Arc, Apple via vLLM-Metal, CPU
SystemsWindows 10 22H2+, macOS 14+, LinuxWindows, macOS, LinuxLinux, Python 3.11-3.14
Model formatsGGUF; MLX on Apple; Safetensors importGGUFHugging Face checkpoints; AWQ, GPTQ, FP8; GGUF experimental
Default address127.0.0.1:11434127.0.0.1:99310.0.0.0:8000 (all interfaces)
Parallel requests by default1 per model4 slotsUp to 256 sequences on cards under 70GB
Multi-user throughputLowest; extra requests queueClose to vLLM at 16 requests in one testHighest at dozens of users
Best forYourself, least setupTinkerers, any GPU or CPUA shared server with many users

What is llama.cpp?

llama.cpp is LLM inference in C/C++: an open-source engine for CPUs and NVIDIA, AMD, Intel and Apple graphics that can split a model between graphics card and system memory. It runs models in GGUF, a single file that packages model weights, tokenizer and metadata. Today one llama command covers both jobs: llama cli to chat in a terminal and llama serve for the server.

What is vLLM?

vLLM describes itself as a fast and easy-to-use library for LLM inference and serving, first developed in UC Berkeley's Sky Computing Lab. Its key idea, PagedAttention, comes from a 2023 paper whose authors reported 2-4 times the throughput of earlier serving systems at the same latency. Its server hosts one model at a time, which suits a dedicated server better than a desktop.

Is Ollama just llama.cpp with a wrapper?

For GGUF models, largely yes. Ollama 0.30, in June 2026, brought GGUF model compatibility through llama.cpp, and Ollama's v0.40.13 code wraps the llama-server binary as a subprocess, pinned to build b11963, up from b11901 in v0.40.12 and b11845 in v0.40.11. Ollama adds the model library, the app, memory sizing and its own API. vLLM is a separate project with its own engine.

Diagram of Ollama vs llama.cpp vs vLLM: the Ollama app wraps llama-server build b11963, while vllm serve uses its own PagedAttention engine, and both draw on GPU memory
Ollama and llama.cpp share one engine, so switching between them changes defaults more than speed; vLLM is a different engine.

So one GGUF file at the same settings should behave much the same in both; the defaults differ. Our LM Studio vs Ollama comparison reaches the same conclusion for LM Studio, which also ships a llama.cpp runtime.

Which models can Ollama, llama.cpp and vLLM run?

The file format decides. GGUF is a binary format designed for fast loading and saving of models, usually already quantized, meaning the weights are stored in fewer bits, such as 4 instead of 16, to save memory. Ollama and llama.cpp run GGUF. vLLM runs Hugging Face checkpoints, the safetensors files listed on a model's page, plus quantized versions in formats such as AWQ, GPTQ and FP8.

AWQ and GPTQ are common 4-bit methods, and FP8 is an 8-bit format that vLLM runs fully on compute capability 8.9 and newer, and as weight-only FP8 on Turing and Ampere. RTX 40 cards are compute capability 8.9 and RTX 30 cards 8.6, so the 3090 gets the weight-only kind.

Can vLLM run GGUF models?

Only just. Its docs call GGUF support highly experimental and under-optimized, and it now needs the separate vllm-gguf-plugin. If your collection is GGUF, keep it in Ollama or llama.cpp and download an AWQ or FP8 build for vLLM.

What fits on a 12GB or 24GB graphics card?

File size is the first test; context is the second. Context length is how much text the model keeps in view; its working memory, the KV cache (key-value cache), grows with every token. Ollama sets context by VRAM: 4K below 24 GiB and 32K from 24 to 48 GiB, so the 4070 starts at 4K and the 3090 at 32K.

Bar chart of model file sizes for llama.cpp, Ollama and vLLM against RTX 4070 12GB and RTX 3090 24GB memory, from Qwen3 8B AWQ at 6.11 GB to Gemma 4 12B 16-bit at 23.9 GB
A 4-bit GGUF 12B model fits the 4070 in Ollama, but vLLM's formats are larger, so the 3090 is the card that makes vLLM practical.

In Ollama's library, Gemma 4 12B at 4-bit is 8.0GB and Gemma 4 31B at 4-bit is 20GB. For vLLM, the official Qwen3 8B AWQ build is 6.11 GB, Red Hat's FP8 Gemma 4 12B is 15 GB, and Google's original 16-bit Gemma 4 12B is 23.9 GB. NVIDIA lists 12GB on the RTX 4070 and 24GB on the RTX 3090.

The upshot: the 4070 runs 12B models in Ollama, but under vLLM I would plan on about 8B at 4-bit, since vLLM budgets 92% of the card, about 11GB, and the KV cache needs part of that. The 3090 has room for FP8 12B-class models. If a model's full context will not fit, vLLM stops with an error that the KV cache needed is larger than the available KV cache memory; lower --max-model-len.

vLLM vs Ollama and llama.cpp: one user or many?

With one person typing, the engines are close. The gap opens when requests overlap. Continuous batching means the engine adds new requests to the batch it is already running, token by token: if 10 requests arrive at roughly the same time, vLLM interleaves their token generation rather than making nine wait. llama-server also has continuous batching enabled by default.

PagedAttention fixes memory. vLLM's launch post said existing systems waste 60% to 80% of memory due to fragmentation and over-reservation of the KV cache. PagedAttention stores each cache in small blocks, like an operating system's memory pages, cutting waste to under 4%, so more conversations fit at once.

Why does vLLM take 92% of my GPU memory?

It reserves its working space up front. The gpu_memory_utilization setting has a default value of 0.92, and vLLM pre-allocates GPU cache using this percentage of memory. On a gaming PC, Windows and open apps already hold some VRAM, and vLLM refuses to start when free memory is below its target. Close games first, or lower the share with --gpu-memory-utilization.

How do Ollama and llama.cpp handle several people?

Ollama's OLLAMA_NUM_PARALLEL sets how many requests each model handles at once, default 1. Raising it costs memory, because required RAM will scale by OLLAMA_NUM_PARALLEL * OLLAMA_CONTEXT_LENGTH: a 2K context with 4 parallel requests becomes an 8K context. Some families, including Qwen 3.5 and Qwen3-VL, are held to one request at a time regardless. Our home AI server guide covers the hardware and power side.

llama.cpp's server uses slots, one conversation each. Left on auto, it uses 4 slots with one shared KV cache, and its docs show llama serve -m model.gguf -c 16384 -np 4 for up to 4 concurrent requests, with the context shared across slots.

What do published benchmarks show?

I know of no published test of all three engines on a gaming card. These are the closest, with their dates and conditions:

Red Hat sells Red Hat AI Inference Server, powered by vLLM, and Gaessler works on llama.cpp, so each is an interested party. Together they suggest a pattern: alone, the engines roughly tie; at a handful of users, llama.cpp with slots stays close; at dozens, vLLM pulls far ahead. For what tokens per second feel like, see our local vs cloud AI speed guide.

Decision diagram for Ollama vs llama.cpp vs vLLM on a gaming PC: just me leads to Ollama, 2 to 4 people to Ollama or llama.cpp with 4 parallel slots, dozens of users or apps to vLLM on Linux or WSL
The number of people asking at the same moment, not the number of people in the house, decides whether vLLM earns its setup time.

How to install and run each engine on Windows 11

Try one engine at a time and quit the others first, since each wants most of the graphics card.

Info card titled three engines, three ports: ollama serve on 127.0.0.1:11434, llama serve on 127.0.0.1:9931 and vllm serve on 0.0.0.0:8000
Three ports mean the engines never collide; note llama.cpp moved from 8080 to 9931 in v0.7.0.

Ollama

  1. Install it. Run the installer from ollama.com/download, or paste irm https://ollama.com/install.ps1 | iex into PowerShell. It runs in the background on localhost:11434; our Ollama install guide covers first errors.
  2. Run a model. Type ollama run gemma4:12b, the command on its library page; the download fits the 4070.
  3. Check the GPU. Run ollama ps: 100% GPU means the model was loaded entirely into the GPU.
  4. Allow parallel requests. Quit Ollama from the taskbar, add OLLAMA_NUM_PARALLEL under Edit environment variables for your account, then start Ollama again from the Start menu.

llama.cpp server

  1. Install it. In PowerShell, run the README's install line, or use winget. The script tries CUDA, then ROCm, Vulkan and CPU, and if it finds an NVIDIA card without the CUDA Toolkit, it suggests winget install Nvidia.CUDA.
  2. Serve a model. llama serve -hf downloads a GGUF file from Hugging Face and serves it at http://localhost:9931.
  3. Open the web UI. Visit that address in your browser to chat, or point other apps at http://localhost:9931/v1.
  4. Size it for the family. Set context and slots, as in the last command below.
irm https://llama.app/install.ps1 | iex
winget install llama.cpp

llama serve -hf ggml-org/gemma-4-e4b-it-GGUF:Q4_0

# up to 4 concurrent requests
llama serve -m model.gguf -c 16384 -np 4

llama serve runs the same server as llama-server, the name in older guides. Those guides also say port 8080, but v0.7.0 made the server listen on port 9931 by default instead of 8080; Docker images keep 8080 inside the container.

How to install vLLM on Windows with Docker

vLLM's docs say it does not support Windows natively, and suggest the Windows Subsystem for Linux (WSL) with a compatible Linux distribution, or community-maintained forks. Docker uses WSL underneath: Docker Desktop passes an NVIDIA GPU to Linux containers only with the WSL 2 backend.

  1. Install WSL. In PowerShell run as administrator, type wsl --install, which installs the Ubuntu distribution, then restart.
  2. Keep your Windows NVIDIA driver. CUDA reaches WSL 2 through it, and NVIDIA warns that users must not install any NVIDIA GPU Linux driver within WSL 2.
  3. Install Docker Desktop. Select Use WSL 2 based engine and turn on WSL integration for Ubuntu.
  4. Test the GPU. Run Docker's n-body sample, the first command below.
  5. Start vLLM. From the Ubuntu terminal, run vLLM's official Docker command, the second command below; it serves on port 8000.
docker run --rm -it --gpus=all nvcr.io/nvidia/k8s/cuda-sample:nbody nbody -gpu -benchmark

docker run --runtime nvidia --gpus all \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=$HF_TOKEN" \
    -p 8000:8000 \
    --ipc=host \
    vllm/vllm-openai:latest \
    --model Qwen/Qwen3-0.6B

That tiny test model loads anywhere; for real use, swap in Qwen/Qwen3-8B-AWQ and add engine arguments after the image tag. If Docker replies unknown or invalid runtime name: nvidia, an error Docker Desktop users have reported, drop --runtime nvidia and keep --gpus all, which is how Docker's own GPU test passes the card.

vLLM on Linux or inside WSL

On a Linux home server, or in Ubuntu on WSL, the quickstart uses uv, a Python package manager you install with curl -LsSf https://astral.sh/uv/install.sh | sh. These commands come from vLLM's quickstart, and the server starts at http://localhost:8000:

uv venv --python 3.12 --seed
source .venv/bin/activate
uv pip install vllm --torch-backend=auto
vllm download-kernels

vllm serve Qwen/Qwen2.5-1.5B-Instruct

Want more detail on llama.cpp? See how to install llama.cpp, including the no-compile options.

Do Ollama, llama.cpp and vLLM all offer an OpenAI-compatible API?

Yes. An OpenAI-compatible API accepts the same requests as OpenAI's, so chat apps and coding tools connect by changing one address. Ollama serves OpenAI endpoints at http://localhost:11434/v1, llama-server offers OpenAI chat completions, responses and embeddings, plus Anthropic Messages, and vLLM implements the OpenAI API protocol.

So swapping engines behind a chat front end is simple. Open WebUI connects to vLLM at http://localhost:8000/v1, or http://host.docker.internal:8000/v1 when Open WebUI runs in Docker. Our Gemma 4 and Open WebUI guide walks through the Ollama side.

Which engine is safest to share on a home network?

Ollama binds 127.0.0.1 port 11434 by default, reachable only from your PC, and its local API does not require authentication. llama.cpp's server also binds 127.0.0.1 by default and accepts --api-key. vLLM differs: with no --host it listens on every network interface, and its --api-key protects only endpoints under /v1, /v2, /inference and /cohere.

vLLM also collects anonymous usage data by default, such as hardware and model type; set VLLM_NO_USAGE_STATS=1 to opt out. On a home server, keep it behind your router, and pass --host 127.0.0.1 if only that machine needs it.

SGLang vs vLLM: do you need SGLang?

SGLang is another open-source serving engine, licensed under Apache 2.0, at v0.5.22, released October 19. Its signature feature, RadixAttention, retains the KV cache for both prompts and generation results in a radix tree, so requests sharing a start, such as a long system prompt, skip that work. Its makers reported up to 5 times higher throughput than Guidance and vLLM in January 2024, but vLLM now lists prefix caching too.

SGLang's install docs target Linux with NVIDIA GPUs, its README lists RTX 30, 40 and 50 series, and its quickstart serves on port 30000. I know of no recent like-for-like test on a consumer card that I trust, so for this reader it can wait.

Common questions about Ollama, llama.cpp and vLLM

Is llama.cpp faster than Ollama?

Usually not by much. Ollama now runs GGUF models through llama.cpp's own server program, so the same file and settings should give similar speed. Differences come from defaults such as context length and parallel slots, and from Ollama pinning a specific llama.cpp build that can trail the newest one.

Is vLLM faster than Ollama?

For one person, not meaningfully. For many people at once, yes. In a 2025 Red Hat test on a data center A100 GPU, vLLM peaked at 793 output tokens per second against 41 for Ollama. Red Hat sells a vLLM-based product, and a gaming card will give different numbers.

Can vLLM run on Windows?

Not natively. The vLLM documentation says to use the Windows Subsystem for Linux with a compatible Linux distribution, or community-maintained forks. The official Docker image also works through Docker Desktop, but GPU passthrough there needs the WSL 2 backend and an NVIDIA graphics card, so it is WSL underneath.

What port does llama-server use?

Since llama.cpp v0.7.0, released in October 2026, the server listens on 127.0.0.1 port 9931 by default instead of 8080. Older guides and builds use 8080, and the official Docker images keep 8080 inside the container. For comparison, Ollama uses port 11434 and vLLM uses port 8000.

Can vLLM run GGUF models?

Only experimentally. The vLLM documentation calls its GGUF support highly experimental and under-optimized, and it now requires the separate vllm-gguf-plugin package. For vLLM, download a Hugging Face build in a supported format such as AWQ, GPTQ or FP8 instead, and keep GGUF files for Ollama or llama.cpp.

Should I use SGLang or vLLM?

For a home server either can work, and both are open source under Apache 2.0 with OpenAI-compatible APIs. SGLang's RadixAttention reuses shared prompt prefixes, which helps agents and long chats, and its docs target Linux with NVIDIA GPUs. Without a trustworthy like-for-like test, pick the one with a ready recipe for your model.

My verdict for a gaming PC: keep Ollama, add vLLM only for a crowd

For the reader with an RTX 4070 or RTX 3090, Ollama remains the right everyday engine: the same llama.cpp core, the least setup and a native Windows app. Move to llama.cpp itself for the newest builds and four slots out of the box. Choose vLLM only for a Linux or WSL home server facing many simultaneous users or apps, and then use the 3090, because 92% of a 12GB card leaves little room once Windows takes its share.

What can wait: SGLang, FP8 tuning and multi-GPU setups. None of them helps a household of four asking questions a few times an hour.

Your next step: set OLLAMA_NUM_PARALLEL to 4, restart Ollama, and have two people ask questions at the same moment while ollama ps still shows 100% GPU. If the wait still annoys them, try the vLLM Docker command above on the 3090 with Qwen/Qwen3-8B-AWQ.

Sources

Keep exploring: Apps · Models · All guides

THE SCOUT LETTER

Good finds.
Less noise.

Useful local AI apps, practical guides and the updates worth your time. A thoughtful note in your inbox.

Free to join. Unsubscribe with one click.