Use HuggingFace Diffusers for Cosmos 3 Generator research, training, and model development. This path loads the full Cosmos 3 checkpoint, including the reasoner path, diffusion generation path, and media tokenizers.

uv venv --python 3.13 --seed --managed-python
source .venv/bin/activate
uv pip install --torch-backend=auto \
  "diffusers @ git+https://github.com/huggingface/diffusers.git" \
  accelerate \
  av \
  cosmos_guardrail \
  huggingface_hub \
  imageio \
  imageio-ffmpeg \
  torch \
  torchvision \
  transformers

--torch-backend=auto lets uv detect your NVIDIA driver and install a matching CUDA build of torch/torchvision. Without it, uv pulls the newest CUDA wheel (currently cu130), which fails on pre-CUDA-13 drivers with The NVIDIA driver on your system is too old and torch.cuda.is_available() returns False. Pin an explicit backend instead if you prefer, e.g. --torch-backend=cu128 for a CUDA 12.8 driver.

A text-to-video run takes a while: the first run downloads Cosmos3-Nano, and diffusion is compute-heavy, running through every inference step before producing output. Long step times are expected, not a hang.

import torch
from diffusers import Cosmos3OmniPipeline
from diffusers.schedulers.scheduling_unipc_multistep import UniPCMultistepScheduler
from diffusers.utils import export_to_video

pipe = Cosmos3OmniPipeline.from_pretrained(
    "nvidia/Cosmos3-Nano",
    torch_dtype=torch.bfloat16,
    device_map="cuda",
)
pipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config, flow_shift=10.0)

result = pipe(
    prompt="A mobile robot navigates a warehouse aisle and stops at a shelf.",
    negative_prompt="",
    image=None,
    num_frames=189,
    height=720,
    width=1280,
    fps=24,
    num_inference_steps=35,
    guidance_scale=6.0,
    enable_sound=False,
    add_resolution_template=False,
    add_duration_template=False,
    generator=torch.Generator(device="cuda").manual_seed(1234),
)

export_to_video(result.video, "cosmos3_t2v.mp4", fps=24, macro_block_size=1)

Diffusers modes:

See the Cosmos 3 Diffusers documentation for runnable examples of each mode.

Use vLLM-Omni for Generator production inference behind an OpenAI-compatible API. This integration loads the full Cosmos 3 checkpoint, including the Qwen3-VL-based reasoner path and the diffusion generation path. For understanding-only tasks that return text, use Reasoner with vLLM instead, which loads only the reasoner.

Compatibility status: Cosmos 3 Generator support has landed in vllm-project/vllm-omni main: text-to-image, text-to-video, and image-to-video (#3454) and video-with-sound (#4073) are merged; action (policy / forward-dynamics) is in review (#4102) and video-to-video is planned. The vllm/vllm-omni:cosmos3 Docker image remains the easiest all-in-one build. For current setup and per-modality usage, see the maintained recipes: Cosmos3-Nano and Cosmos3-Super.

Start the server from the Docker image (all modalities). Mount any directory that contains local media or action files you want the server to read.

docker run --runtime nvidia --gpus all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -v "$(pwd):/workspace" \
  -p 8000:8000 \
  --ipc=host \
  vllm/vllm-omni:cosmos3 \
  vllm serve nvidia/Cosmos3-Nano \
  --omni \
  --model-class-name Cosmos3OmniDiffusersPipeline \
  --allowed-local-media-path / \
  --port 8000 \
  --init-timeout 1800

Cosmos3 checkpoints can exceed the default server init timeout; use --init-timeout 1800 on every vllm serve command in this section.

vLLM-Omni prints Application startup complete. when the API is ready.

For nvidia/Cosmos3-Super (the larger 64B model), split weights across GPUs and optionally offload layers to reduce peak memory: --tensor-parallel-size splits model weights across multiple GPUs, and --enable-layerwise-offload offloads transformer blocks between CPU and GPU with a latency tradeoff and extra CPU RAM use. For example, on four GPUs, add --tensor-parallel-size 4 --enable-layerwise-offload --init-timeout 1800 to the vllm serve command.

Additional parallelism options:

When combining parallelism options, ensure the server has enough GPUs for the product of the enabled degrees (tensor_parallel_size × cfg_parallel_size × ulysses_degree).

To install vLLM-Omni from main instead of using the Docker image (text-to-image, text-to-video, image-to-video, and video-with-sound are merged there; see the Cosmos3-Nano and Cosmos3-Super recipes for per-modality usage), create a venv and install, choosing the CUDA build that matches your driver:

uv venv --python 3.13 --seed --managed-python
source .venv/bin/activate
# CUDA 13 driver:
uv pip install --torch-backend=cu130 \
  "vllm-omni @ git+https://github.com/vllm-project/vllm-omni.git@main"
# CUDA 12.8 driver:
# uv pip install --torch-backend=cu128 \
#   "vllm-omni @ git+https://github.com/vllm-project/vllm-omni.git@main"

Then run vllm serve nvidia/Cosmos3-Nano --omni --model-class-name Cosmos3OmniDiffusersPipeline --allowed-local-media-path / --port 8000 --init-timeout 1800 directly, without the docker run ... vllm/vllm-omni:cosmos3 wrapper.

Vision endpoints:

Action modes use Cosmos 3 as a world model: they condition on an embodiment (domain_name) and exchange video and action sequences. Policy and inverse dynamics return a predicted action chunk, so send those through the asynchronous POST /v1/videos job and read the action data from the completed result; forward dynamics returns only video and can use synchronous POST /v1/videos/sync.

Pass embodiment settings through extra_params: action_mode, domain_name (for example bridge_orig_lerobot, av, or camera_pose), raw_action_dim, and action_chunk_size. Forward dynamics also takes an action_path pointing at an action file the server can read, so start the server with --allowed-local-media-path covering that file (for Docker, mount the file and pass the container-visible path). For the full set of robot, autonomous-vehicle, and camera-pose variants, see the Cosmos 3 vLLM-Omni recipes.

Example video request:

curl -sS -X POST http://localhost:8000/v1/videos/sync \
  --form-string "prompt=A small warehouse robot moves a blue box across a clean floor." \
  --form-string "negative_prompt=blurry, distorted, low quality" \
  --form-string "size=1280x720" \
  --form-string "num_frames=189" \
  --form-string "fps=24" \
  --form-string "num_inference_steps=35" \
  --form-string "guidance_scale=6.0" \
  --form-string "flow_shift=10.0" \
  --form-string "seed=0" \
  --form-string 'extra_params={"use_resolution_template":false,"use_duration_template":false,"guardrails":true}' \
  -o cosmos3_t2v_output.mp4

Use --form-string for text fields (prompt, negative_prompt, extra_params) rather than -F: with -F, curl treats ; as a content-type separator and silently truncates any value that contains one.

Common request fields (the image endpoint follows the Image Generation API, and the video endpoints follow the Videos API):

Disabling guardrails: Cosmos 3 ships safety guardrails that screen prompts and blur faces in generated output. Disable them per request by adding guardrails: false to extra_params:

curl -sS -X POST http://localhost:8000/v1/videos/sync \
  --form-string "prompt=A small warehouse robot moves a blue box across a clean floor." \
  --form-string 'extra_params={"guardrails":false,"use_resolution_template":false,"use_duration_template":false}' \
  -o cosmos3_t2v.mp4

To disable guardrails server-wide so the guardrail models are never loaded (per-request overrides then cannot turn them back on), pass a deploy config — a future release replaces this with a dedicated --cosmos3-no-guardrails flag:

# no_guardrails.yaml
async_chunk: false
stages:
  - stage_id: 0
    max_num_seqs: 1
    enforce_eager: true
    trust_remote_code: true
    model_class_name: Cosmos3OmniDiffusersPipeline
    model_config:
      guardrails: false
      offload_guardrail_models: false

vllm serve nvidia/Cosmos3-Nano --omni \
  --model-class-name Cosmos3OmniDiffusersPipeline \
  --deploy-config no_guardrails.yaml \
  --port 8000 \
  --init-timeout 1800

References:

• Cosmos 3 vLLM-Omni recipes
• vLLM-Omni Videos API
• vLLM-Omni Image Generation API

uv venv --python 3.13 --seed --managed-python
source .venv/bin/activate
uv pip install --torch-backend=cu130 "vllm==0.21.0" \
  "vllm-cosmos3 @ git+https://github.com/NVIDIA/cosmos-framework.git#subdirectory=packages/vllm-cosmos3"

The vLLM version and the torch backend are paired: use --torch-backend=cu130 "vllm==0.21.0" for a CUDA 13 driver, or --torch-backend=cu128 "vllm==0.19.1" for CUDA 12.8. vLLM does not publish wheels for every CUDA minor version, so --torch-backend=auto is not reliable here — pick the pair that matches your driver.

vllm serve nvidia/Cosmos3-Nano \
  --hf-overrides '{"architectures": ["Cosmos3ReasonerForConditionalGeneration"]}' \
  --async-scheduling \
  --allowed-local-media-path / \
  --port 8000

For notebook launch commands (Cosmos3-Super on four GPUs, media-path defaults, and full flag sets), see cookbooks/cosmos3/README.md — Start the server.

If your vLLM build reports that DeepGEMM is unavailable, disable it before starting the server:

export VLLM_USE_DEEP_GEMM=0

Configuration notes:

Use the Cosmos 3 Reasoner NIM for the fastest path to a production-grade, OpenAI-compatible Reasoner endpoint. NIM ships a prebuilt, optimized container so you skip the vLLM dependency and CUDA-pairing setup above; it serves text outputs from text, image, and video inputs.

You can try it interactively in your browser on the cosmos3-nano-reasoner build page — that playground is powered by this same NIM. See the Cosmos Reason 3 NIM API reference for the full request reference.

The container serves two sizes, selected with NIM_MODEL_SIZE:

Launch the NIM container (Nano shown; set -e NIM_MODEL_SIZE=super for Super). NGC_API_KEY must be set in your environment first — generate one from NGC and log Docker in to nvcr.io once (docker login nvcr.io, username $oauthtoken, password = your key).

export CONTAINER_NAME="nvidia-cosmos3-reasoner"
export IMG_NAME="nvcr.io/nim/nvidia/cosmos3-reasoner:1.7.0"
export LOCAL_NIM_CACHE=~/.cache/nim
mkdir -p "$LOCAL_NIM_CACHE"

docker run -it --rm --name=$CONTAINER_NAME \
  --runtime=nvidia \
  --gpus all \
  --shm-size=32GB \
  -e NGC_API_KEY=$NGC_API_KEY \
  -e NIM_MODEL_SIZE=nano \
  -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
  -u $(id -u) \
  -p 8000:8000 \
  $IMG_NAME

The OpenAI-compatible API is then available at http://127.0.0.1:8000/v1. Query it with curl:

IMAGE_DATA_URI="data:image/jpeg;base64,$(base64 -w 0 cookbooks/cosmos3/reasoner/assets/robot_153.jpg)"

curl -X POST 'http://127.0.0.1:8000/v1/chat/completions' \
  -H 'Accept: application/json' \
  -H 'Content-Type: application/json' \
  --data-binary @- <<JSON
{
    "model": "nvidia/cosmos3-nano-reasoner",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": [
        {"type": "image_url", "image_url": {"url": "$IMAGE_DATA_URI"}},
        {"type": "text", "text": "Describe what is happening in this image in one sentence."}
      ]}
    ],
    "max_tokens": 256,
    "stream": false
  }
JSON

Or with the OpenAI Python client:

from openai import OpenAI

# The container exposes the OpenAI-compatible API locally; the api_key is unused.
client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="not-used")

response = client.chat.completions.create(
    model="nvidia/cosmos3-nano-reasoner",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {
            "role": "user",
            "content": [
                {"type": "video_url", "video_url": {"url": "https://download.samplelib.com/mp4/sample-5s.mp4"}},
                {"type": "text", "text": "List the notable events with approximate timestamps."},
            ],
        },
    ],
    max_tokens=256,
    stream=False,
    extra_body={"media_io_kwargs": {"video": {"fps": 4.0}}},
)
print(response.choices[0].message.content)

Inputs and request notes:

References:

• Cosmos Reason 3 NIM API reference
• cosmos3-reasoner NGC container
• cosmos3-nano-reasoner build page