vLLM not working? 8 errors, the exact message and the fix

When vLLM fails, the message usually names the culprit: no ready-made build for Windows or macOS, a card it cannot see, a driver too old for CUDA 13, memory already taken, a cache that does not fit, weights too big, a model it cannot fetch, or a busy port. Here are the 8, with the exact text and the fix.

vLLM is an engine built to serve models to many people at once, and getting it running on your own NVIDIA card feels like borrowing a data centre’s toolkit for the weekend. The flip side is that it says no more often than Ollama or LM Studio, and in longer sentences. Every message below was checked on 15 October 2026 against the code of vLLM 0.31.0, still the latest stable version that day, and against the PyTorch 2.13.0 it ships with. On 19 October 2026 we compared them with the code of 0.32.0, the stable version since 18 October 2026: the messages that come from vLLM’s own code are word for word the same, and the few details that did change are marked below. The pip and Hugging Face messages we triggered ourselves; the rest come straight from that code and its issue tracker. Installing it and the first model are covered in our vLLM setup guide; this page picks up where that one ends.

vLLMRuntime sheet→

Windows and macOS: “No matching distribution found for vllm”

The first wall is often the installer itself:

ERROR: Could not find a version that satisfies the requirement vllm==0.31.0 (from versions: none)
ERROR: No matching distribution found for vllm==0.31.0

We got exactly that by asking pip for ready-made builds only (--only-binary=:all:) of vLLM 0.31.0, as if from Windows and again as if from an Apple silicon Mac. The reason is simple: on PyPI, 0.31.0 only publishes ready-made builds for Linux, on x86_64 and on ARM, and 0.32.0 publishes the same 2. A plain pip install vllm goes one step further: it downloads the source package (44.5 MB in 0.32.0, 43.4 MB in 0.31.0) and tries to build it, which on Windows is not a way to a working vLLM. vLLM’s own documentation says it plainly: “vLLM does not support Windows natively”. On Windows the way in is WSL 2 with an Ubuntu inside. One detail from NVIDIA that saves an evening: do not install any Linux display driver inside WSL. The normal Windows driver already hands the card to WSL, and nvidia-smi run inside Ubuntu should list it before you go any further.

On a Mac, vLLM runs on the CPU only, as experimental support, in FP32 and FP16, and the documentation tells you to build it from source. The 0.31.0 release does attach a ready-made CPU build for Apple silicon and Python 3.12, which the documentation says does not exist yet, and 0.32.0 attaches one that is no longer tied to Python 3.12; we have not tested either. For the Mac’s GPU, vLLM points to vLLM-Metal, a community plugin that runs on MLX. If what you want is a fast model on a Mac, our MLX setup guide is the shorter road.

One more pip message worth knowing: ask for vllm==0.32.0 on Python 3.10, or for any ready-made build on 3.9, and you get ERROR: Ignored the following versions that require a different python version, followed by a list that ends in 0.32.0 Requires-Python <3.15,>=3.11 (that is how pip 25.1 printed it for us on 21 October 2026, asked as if from Python 3.10). Since 0.32.0, vLLM’s guide asks for Python 3.11 to 3.14. Careful with 3.10: there, a plain pip install vllm does not fail, it quietly installs 0.31.0: no warning, just the version number in pip’s last line, and in vllm --version.

“Failed to infer device type” or “No CUDA GPUs are available”: vLLM cannot see the card

This one appears before any model loads:

RuntimeError: Failed to infer device type, please set the environment variable `VLLM_LOGGING_LEVEL=DEBUG` to turn on verbose logging to help debug the issue.

vLLM looks for a platform when it starts: an NVIDIA card (through NVIDIA’s management library), AMD, Intel and a few others. The CPU only counts if you installed a CPU build, set VLLM_TARGET_DEVICE=cpu, or you are on a Mac. The normal build from PyPI, on a machine where no NVIDIA card is visible, finds nothing, and this is the result. “Visible” is the key word: a Docker container started without --gpus all, a WSL where the card does not show up, or a laptop without an NVIDIA card all end the same way.

Its cousin comes from PyTorch: RuntimeError: No CUDA GPUs are available. It means PyTorch counted 0 cards. vLLM’s own code uses that very text as the sign that CUDA_VISIBLE_DEVICES is empty, so check that variable first: set to an empty value, or to a card number that does not exist, it hides every card from the process.

“The NVIDIA driver on your system is too old”, or “no kernel image is available”

Here vLLM did find the card, and PyTorch stops right after:

RuntimeError: The NVIDIA driver on your system is too old (found version 12080). Please update your GPU driver by downloading and installing a new version from the URL: http://www.nvidia.com/Download/index.aspx Alternatively, go to: https://pytorch.org to install a PyTorch version that has been compiled with your version of the CUDA driver.

The number is the newest CUDA your driver supports, written as 1000 × major + 10 × minor: 12080 means CUDA 12.8. That is the catch, because the build on PyPI, in 0.31.0 as in 0.32.0, is a CUDA 13.0 build, as its release notes say (the installation page at that tag still says 12.9). And NVIDIA’s rule is that CUDA 13 programs need a driver of version 580 or newer. So you have 2 ways out. The clean one: update the driver to 580 or newer, then check that nvidia-smi shows “CUDA Version: 13.0” or higher in its top right corner. If you cannot update, vLLM publishes a CUDA 12.9 build of the same version on its releases page, and a vllm/vllm-openai:v0.32.0-cu129 Docker image (v0.31.0-cu129 for 0.31.0); its installation guide shows the command for picking a build by CUDA version.

The other message in this family has nothing to do with the driver:

torch.AcceleratorError: CUDA error: no kernel image is available for execution on the device

It means the code was not compiled for your card’s architecture: either the card is too old, or the build or image you use leaves it out. vLLM asks for an NVIDIA card with compute capability 7.5 or higher: T4, RTX 20 series and newer. The NVIDIA cards in our catalogue are all above that: the RTX 3060 and 3090 are 8.6, and the RTX 4060, 4070 and 4090 are 8.9. On an older card, PyTorch usually warns first with … with CUDA capability sm_XX is not compatible with the current PyTorch installation. If you see that line, no setting will fix it: the card is below the bar.

“Free memory on device … is less than desired GPU memory utilization”

This is vLLM being greedy on purpose. When it starts, it wants a fixed share of the card for itself, 0.92 of its total memory by default, and it checks that share against the memory that is free at that moment:

ValueError: Free memory on device … (X/Y GiB) on startup is less than desired GPU memory utilization (0.92, Z GiB). Decrease GPU memory utilization or reduce GPU memory used by other processes.

X is what was free, Y the card’s total and Z the 92% vLLM wanted. Whatever is already on the card counts against you: your desktop, a browser using the GPU, Ollama with a model still loaded, or a second vLLM. The fix is the one the message gives. Lower the share with --gpu-memory-utilization 0.85, or free the card first: nvidia-smi lists what is using memory. The value is a limit per process, so 2 vLLM servers on 1 card each need their own, smaller value: vLLM’s documentation gives 0.5 each as its example. In this short video, one developer keeps 2 models loaded on a single 96 GB card, one at 0.7 and the other at 0.2, leaving 10% free:

“To serve at least one request with the model’s max seq len”: the KV cache does not fit

This is the most “vLLM” error of them all, and once you can read it, it is the most helpful:

ValueError: To serve at least one request with the model's max seq len (40960), (5.62 GiB KV cache is needed, which is larger than the available KV cache memory (Y GiB). Based on the available memory, the estimated maximum model length is Z. Try increasing `gpu_memory_utilization` (which also controls CPU memory on the CPU backend) or decreasing `max_model_len` when initializing the engine. See https://docs.vllm.ai/en/latest/configuration/conserving_memory/ for more details.

The KV cache is the memory where the model keeps the conversation it is working on. vLLM does not grow it as you go: unless you say otherwise, it plans for the longest context the model accepts, taken from the model’s own configuration, and refuses to start if one full-length request does not fit. If nothing at all is left after the weights, the message is shorter: No available memory for the cache blocks. And if you are on an older vLLM, the same problem reads The model's max seq len (4096) is larger than the maximum number of tokens that can be stored in KV cache (3664).

The useful part is Z: vLLM has already calculated the longest context that would fit. Use it, or round it down:

vllm serve Qwen/Qwen3-8B-AWQ --max-model-len 16384

Mind the capital letter if you use a suffix: 16K is 16,384, while 16k is 16,000. With --max-model-len auto (or -1), vLLM picks the longest context that fits by itself and says so in the log: Auto-fit max_model_len: reduced from … to … to fit in available GPU memory (… GiB available for KV cache). And if you ask for more than the model allows, it stops with User-specified max_model_len (…) is greater than the derived max_model_len (…).

How much context fits in each GiB depends on the model’s architecture, not on its size. Our numbers, from the same formula vLLM uses for full-attention layers, with a 16-bit cache:

Model	Cache per token	Tokens per GiB of cache	Default context	Cache at the default
Qwen3 8B	144 KiB	7,281	40,960	5.625 GiB
Qwen3 4B 2507	144 KiB	7,281	262,144	36.0 GiB
Granite 4.2 8B	160 KiB	6,553	131,072	20.0 GiB
Qwen3 Coder 30B	96 KiB	10,922	262,144	24.0 GiB

Calculated by Local AI Scope on 15 October 2026: 2 × layers × KV heads × head size × 2 bytes per token, from each model’s configuration, the data behind our model finder. Models with sliding-window or linear-attention layers follow other rules and are left out.

PC with an RTX 3060 12 GBHardware sheet→

Another lever: --kv-cache-dtype fp8 stores the cache in 8 bits instead of 16, half the memory per token. That buys context, and it is a trade-off we have not measured, so try it on your own prompts. If you want the bigger picture, vLLM’s own talk at PyTorch Conference 2025 goes into how it manages the KV cache from 10:08:

“CUDA out of memory” while loading: the weights do not fit

When the model itself is too big, vLLM does not get as far as the cache. PyTorch stops during loading:

torch.OutOfMemoryError: CUDA out of memory. Tried to allocate … GPU 0 has a total capacity of … of which … is free. …

The usual cause is a model in its original 16-bit format. vLLM loads those safetensors files as they are, and Qwen3 8B in BF16 takes 15.26 GiB: on a 12 GB card it does not fit before the first token. The fixes, from vLLM’s guide on saving memory, in the order we would try them:

• A quantized version. The same model in AWQ is 5.68 GiB, and in FP8, 8.79 GiB. Look for those versions on Hugging Face, often published by the model’s own makers.
• Less context and fewer requests at once: --max-model-len and --max-num-seqs.
• No CUDA graphs: --enforce-eager skips them and frees their memory, at some cost in speed.
• More than 1 card: --tensor-parallel-size 2 splits the model across 2 GPUs.

One warning for those coming from Ollama: the sizes in our model finder are for GGUF files, such as Q4_K_M, which vLLM does not load out of the box. For vLLM they are a guide, not the same number.

“Invalid repository ID”, gated models and “trust_remote_code”

vLLM fetches models from Hugging Face, and 3 kinds of message come from there. A single typo in the name gives this, with Hugging Face’s Repository Not Found underneath:

ValueError: Invalid repository ID or local directory specified: 'Qwen/Qwen3-8BB'.
Please verify the following requirements:
1. Provide a valid Hugging Face repository ID.
2. Specify a local directory that contains a recognized configuration file.
   - For Hugging Face models: ensure the presence of a 'config.json'.
   - For Mistral models: ensure the presence of a 'params.json'.

Check the name letter by letter. If the name is right but the model makes you accept its conditions first, as Meta’s Llama models do, the text further up in the traceback is this one, which we got by asking for Llama 3.1 8B without logging in:

Cannot access gated repo for url https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct/resolve/main/config.json.
Access to model meta-llama/Llama-3.1-8B-Instruct is restricted. You must have access to it and be authenticated to access it. Please log in.

Accept the conditions on the model’s page with your account, then log in once with hf auth login. vLLM reads the token it saves, and it also takes one with --hf-token or the HF_TOKEN variable.

The third is a model that brings its own Python code. With the transformers version vLLM 0.31.0 installs (we ran 5.17.0, which 0.32.0 also accepts), it reads:

ValueError: The repository THUDM/chatglm3-6b contains custom code which must be executed to correctly load the model. You can inspect the repository content at https://hf.co/THUDM/chatglm3-6b .
 You can inspect the repository content at https://hf.co/THUDM/chatglm3-6b.
Please pass the argument `trust_remote_code=True` to allow custom code to be run.

On the command line, the switch is --trust-remote-code. And if the model’s architecture is simply not in vLLM, the message is Model architectures ['…'] are not supported for now. Supported architectures: …, or, for some old ones, … was supported in vLLM until v0.23.0, and is not supported anymore. GGUF files are a separate case: their support now lives in a plugin, vllm-gguf-plugin, which vLLM’s documentation still calls “highly experimental”.

Port 8000 “Address already in use”, and “Engine core initialization failed”

vllm serve takes its port before it even starts the engine, so a busy port fails in seconds, without loading anything:

OSError: [Errno 98] Address already in use

By default that port is 8000, on every network interface. Something else is listening there: another vLLM you forgot about, or any other server. Find out who has it, or use another port:

ss -ltnp | grep :8000
vllm serve Qwen/Qwen3-8B-AWQ --port 8001

The last message is the one people paste the most, and it is not a cause at all:

RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}

vLLM runs the model in a separate process, and when that process dies while starting, this line is just the wrapper. “See root cause above” means exactly that: scroll up the log to the first error, and it will be one of the ones on this page, very often the memory ones. One report in vLLM’s tracker shows it on top of an [Errno 98] Address already in use from one of its workers. And once it starts, close what it opens to the network: our setup guide explains why it listens on 0.0.0.0, with no key.

Messages and defaults from the code of vLLM 0.31.0 and PyTorch 2.13.0, vLLM’s installation and memory guides at that tag, NVIDIA’s CUDA documentation, Hugging Face, vLLM’s issue tracker, and our own runs of pip, transformers 5.17.0 and huggingface_hub 1.33.0, on 15 October 2026. vLLM’s messages compared again with the code of 0.32.0 on 19 October 2026, and the Python message of pip, on 21 October 2026.

Running Ollama, LM Studio or MLX as well? Their errors are different, and so are the fixes: Ollama not working? 9 errors, the exact message and the fix, LM Studio not working? 7 errors, the exact message and the fix and MLX not working? 7 errors, the exact message and the fix. And for what vLLM sends out while all this works, see its sheet: vLLM: what it sends home and whether you can audit it.