Run an LLM with Docker or Podman.

Before you pull weights

1. Open the developer's model card and copy the exact official model ID.
2. Confirm the weight license permits your use and distribution model.
3. Check current vLLM model support, required Transformers version, quantization, and GPU memory.
4. Keep registry and Hugging Face credentials outside images and source control.

Docker

docker run --rm --gpus all --ipc=host -p 8000:8000 \
  -v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
  vllm/vllm-openai:v0.25.1 --model <OFFICIAL_MODEL_ID>

Podman

podman run --rm --device nvidia.com/gpu=all --ipc=host -p 8000:8000 \
  -v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
  docker.io/vllm/vllm-openai:v0.25.1 --model <OFFICIAL_MODEL_ID>

Podman uses a fully qualified image name to avoid ambiguous short-name resolution. GPU device syntax can differ by host and container-toolkit setup.

Call the local server

curl http://localhost:8000/v1/models
# Use the served model ID with an OpenAI-compatible client after the health check succeeds.

Operational limits

• Replace <OFFICIAL_MODEL_ID> only with an identifier from the model developer's official model card.
• Downloadable weights do not prove vLLM compatibility; verify the current supported-model documentation before pulling large artifacts.
• The reviewed vLLM image is pinned for reproducibility. Re-review compatibility and release notes before changing the tag.
• The mounted Hugging Face cache is visible to the container. If gated weights require authentication, use a scoped read token in the host cache, restrict host-file and daemon access, and revoke it when no longer needed.
• GPU device flags, shared memory, rootless Podman, SELinux labels, quantization, and tensor parallelism are host-specific.
• Review the model license before commercial use or redistribution. Never bake access tokens into images or source control.