vLLM: Scale-out disaggregated multimodal transport trusts caller-supplied features

Affected code

Links pinned to the confirmed commit 752a3a504485 (v0.25.1).

Shared entry point and control surface for all five sites:

• POST /inference/v1/generate route: vllm/entrypoints/scale_out/token_in_token_out/api_router.py#L46-L75.
• Public features schema (kwargs_data, mm_hashes, mm_placeholders): vllm/entrypoints/scale_out/token_in_token_out/protocol.py#L42-L63.
• ServingTokens.serve_tokens() copies the decoded geometry and hashes into engine structures without rebinding to the renderer schema: vllm/entrypoints/scale_out/token_in_token_out/serving.py#L145-L172.
• The scale-out routers are registered by default on generate-capable servers and /inference is treated as an ordinary auth-guarded prefix: vllm/entrypoints/openai/api_server.py#L217-L219.

Site 1 — forged Qwen grid geometry (engine-fatal DoS). The decoded image_grid_thw is never rebound to the rendered pixel-tensor element count.

• Reconstruction with no model-specific geometry invariant: serving.py#L147-L172.
• Qwen2VisionTransformer.prepare_encoder_metadata() derives RoPE tables, cu_seqlens, and the FlashAttention max_seqlen from the caller-supplied grid: qwen2_vl.py#L658-L712, consumed in forward() (#L713-L751).
• _process_image_input() computes image_embeds.split(sizes) from the same untrusted metadata: qwen2_vl.py#L1348-L1369; M-RoPE positions at qwen2_vl.py#L1223.

The only shape check is assert grid_thw.ndim == 2; the split sizes and the vision-encoder call are then derived directly from the caller-supplied grid, with no cross-check against the pixel-tensor row count:

# vllm/model_executor/models/qwen2_vl.py Lines 1348-1369
    def _process_image_input(
        self, image_input: Qwen2VLImageInputs
    ) -> tuple[torch.Tensor, ...]:
        grid_thw = image_input["image_grid_thw"]
        assert grid_thw.ndim == 2

        if image_input["type"] == "image_embeds":
            image_embeds = image_input["image_embeds"]
        else:
            pixel_values = image_input["pixel_values"]

            if self.use_data_parallel:
                return run_dp_sharded_mrope_vision_model(
                    self.visual, pixel_values, grid_thw.tolist(), rope_type="rope_3d"
                )
            else:
                image_embeds = self.visual(pixel_values, grid_thw=grid_thw)

        # Split concatenated embeddings for each image item.
        merge_size = self.visual.spatial_merge_size
        sizes = (grid_thw.prod(-1) // merge_size // merge_size).tolist()
        return image_embeds.split(sizes)

Site 2 — wire-selected field-processor type confusion (engine-fatal DoS). MsgpackDecoder._decode_mm_field_elem() trusts a wire-selected field-factory name and constructs the internal field processor directly from caller data.

• vllm/v1/serial_utils.py#L440-L454 reads factory_meth_name, factory_kw = obj["field"] and calls getattr(MultiModalFieldConfig, factory_meth_name).
• Qwen2-VL field/schema contract: qwen2_vl.py#L763-L791 and Qwen2VLImagePixelInputs (#L119-L144); parse/validate at qwen2_vl.py#L1300.
• Rank mismatch raised post-admission: vllm/utils/tensor_schema.py#L155-L171, reached from TensorSchema.__init__ → validate() (#L63).

Site 3 — non-positive placeholder length (engine-fatal DoS via reachable assert). PlaceholderRangeInfo{offset,length} is accepted as unconstrained integers and copied verbatim into the engine's PlaceholderRange.

• Schema: protocol.py#L28-L35.
• Copied into PlaceholderRange: serving.py#L147-L153.
• The only guard is an upper bound on embed count (num_embeds > mm_encoder_cache_size), with no non-positive check: vllm/v1/engine/input_processor.py#L459-L464; raw length returned by vllm/multimodal/inputs.py#L152-L154; window selection assumes non-empty ranges at vllm/multimodal/utils.py#L114-L134.
• Sink: assert start_idx < end_idx at vllm/v1/worker/gpu/mm/encoder_runner.py#L114 (duplicated at vllm/v1/worker/gpu_model_runner.py#L3192); turned into a fatal shutdown by EngineCore's uncaught-exception path at vllm/v1/engine/core.py#L1229-L1233.

A length of 0 makes num_encoder_tokens == 0, so end_idx collapses to 0 and the bare assert fires inside the engine worker:

# vllm/v1/worker/gpu/mm/encoder_runner.py Lines 108-114
                pos_info = mm_feature.mm_position
                start_pos = pos_info.offset
                num_encoder_tokens = pos_info.length

                start_idx = max(cur_query_start - start_pos, 0)
                end_idx = min(cur_query_end - start_pos, num_encoder_tokens)
                assert start_idx < end_idx

Site 4 — cache hash not bound to payload (integrity / disclosure). features.mm_hashes (the cache key) and kwargs_data (the tensor) are independent fields with no origin or integrity binding.

• Schema exposing the caller-controlled hash and the None = resolve-from-cache semantics: protocol.py#L42-L63.
• Handler forwards the caller's hashes unchanged into mm_input(...): serving.py#L164-L170.
• Input processing copies the caller hash into the feature identifier with only a string-type check: vllm/v1/engine/input_processor.py#L165-L181.
• Sink — the receiver cache returns the cached tensor solely by that key (cache_key = feature.mm_hash or feature.identifier): vllm/multimodal/cache.py#L602-L607.

The cache key is the caller-supplied hash with no verification against the tensor bytes, so a forged mm_hash both stores under and reads back another request's slot:

# vllm/multimodal/cache.py Lines 601-607
        for feature in mm_features:
            cache_key = feature.mm_hash or feature.identifier
            self.touch_receiver_cache_item(cache_key, feature.data)

        for feature in mm_features:
            cache_key = feature.mm_hash or feature.identifier
            feature.data = self.get_and_update_item(feature.data, cache_key)
        return mm_features

Site 5 — dropped sparse placeholder mask (transport integrity loss). The render path serializes placeholders as only offset/length, so models relying on sparse is_embed masks lose the mask during render-to-generate replay.

• ServingRender._extract_mm_features() builds each PlaceholderRangeInfo(offset=p.offset, length=p.length), discarding is_embed: vllm/entrypoints/scale_out/render/serving.py#L212-L229.
• Transport schema carries no field for the mask: protocol.py#L28.
• ServingTokens.serve_tokens() reconstructs a dense PlaceholderRange regardless of the original: serving.py#L148-L152.

Impact

A single authenticated request to a scale-out multimodal deployment can:

• Crash the shared EngineCore process (sites 1, 2, 3), taking the served model down for every tenant (/health → 503). Availability-only; no code execution or data disclosure demonstrated for these sites.
• Silently poison or read back another request's shared encoder-cache state (site 4) — an integrity/disclosure primitive. Attack complexity is High because the attacker must know or induce the victim's content hash; there is no availability impact for this site.
• Corrupt backend-visible placeholder semantics across the render-to-generate boundary (site 5) for models that depend on sparse is_embed masks.

The forged multimodal payload is small; only the trust in its self-declared geometry/identity is the defect.

Suggested Fix

On the scale-out path, do not trust caller-supplied multimodal state as renderer-produced. After decoding features, rebind and validate the reconstructed MultiModalKwargsItem against the active model's renderer contract at the HTTP boundary:

1. Reject any request whose decoded grid geometry is inconsistent with the rendered pixel-tensor element count and declared placeholder span, before it reaches prepare_encoder_metadata() (site 1).
2. Rebind each field's processor type to the schema the active model's renderer declares (or reject if it differs), instead of reconstructing internal field processors from wire-selected factory names (site 2).
3. Reject any PlaceholderRangeInfo with length <= 0 (or out-of-range offset) with a request-scoped 4xx, and convert the encoder-runner invariant into a checked, request-scoped error rather than a process-fatal assert (site 3).
4. Recompute or verify the content hash for submitted kwargs_data before using it as a cache key, and namespace receiver-cache keys to a server-generated or principal scope, refusing cache-reads for hashes the caller did not legitimately produce (site 4).
5. Serialize is_embed in PlaceholderRangeInfo, validate its length against the placeholder span, and reconstruct it on replay (site 5).

Site 1 — validate grid geometry before the vision encoder. Replace the bare assert grid_thw.ndim == 2 in _process_image_input()/_process_video_input() with a shared helper that recomputes the split sizes and rejects a grid whose patch-row count does not match the pixel tensor (and rejects non-positive / non-merge-divisible dims), so the mismatch never reaches image_embeds.split():

# vllm/model_executor/models/qwen2_vl.py — _process_image_input()
-        grid_thw = image_input["image_grid_thw"]
-        assert grid_thw.ndim == 2
+        grid_thw = image_input["image_grid_thw"]
+        input_type = image_input["type"]
+        input_tensor = (
+            image_input["image_embeds"]
+            if input_type == "image_embeds"
+            else image_input["pixel_values"]
+        )
+        sizes = _validate_qwen2_vl_input_geometry(
+            modality="image",
+            input_type=input_type,
+            input_tensor=input_tensor,
+            grid_thw=grid_thw,
+            spatial_merge_size=self.visual.spatial_merge_size,
+        )
...
-        # Split concatenated embeddings for each image item.
-        merge_size = self.visual.spatial_merge_size
-        sizes = (grid_thw.prod(-1) // merge_size // merge_size).tolist()
         return image_embeds.split(sizes)

where the helper raises before the encoder runs:

# vllm/model_executor/models/qwen2_vl.py — new _validate_qwen2_vl_input_geometry()
        if t <= 0 or h <= 0 or w <= 0:
            raise ValueError(f"{modality} grid_thw row {index} must be positive ...")
        if h % spatial_merge_size != 0 or w % spatial_merge_size != 0:
            raise ValueError(f"{modality} grid_thw row {index} must be divisible ...")
        ...
        if actual_rows != expected_rows:
            raise ValueError(
                f"{modality} {row_kind} do not match grid_thw: "
                f"expected {expected_rows}, got {actual_rows}."
            )

Site 3 — constrain the placeholder schema. Make PlaceholderRangeInfo reject non-positive lengths and negative offsets at the Pydantic boundary (plus parallel-length, non-overlapping, and within-prompt validators), turning the process-fatal assert into a request-scoped 422:

# vllm/entrypoints/scale_out/token_in_token_out/protocol.py — PlaceholderRangeInfo
-    offset: int
-    length: int
+    offset: int = Field(ge=0)
+    length: int = Field(gt=0)

Site 4 — bind the cache key to the payload. Derive each cache key by hashing the submitted serialized tensor (ignoring the caller's mm_hashes) and refuse cache-only reads, so a forged hash can neither poison nor read a victim's slot:

# vllm/entrypoints/scale_out/token_in_token_out/serving.py — serve_tokens()
+            mm_hashes = _bind_mm_hashes_to_kwargs_data(
+                features.mm_hashes, features.kwargs_data,
+            )
             engine_input = mm_input(
                 prompt_token_ids=request.token_ids,
                 mm_kwargs=MultiModalKwargsItems(mm_kwargs),
-                mm_hashes=features.mm_hashes,
+                mm_hashes=mm_hashes,
                 mm_placeholders=mm_placeholders,
                 cache_salt=request.cache_salt,
             )

where _bind_mm_hashes_to_kwargs_data() raises on kwargs_data is None (cache-only read) and derives sha256(modality || "\0" || serialized_item) per item. Site 2 applies the same rebind-to-declared-schema pattern in mm_serde.py (passing modality + mm_processor into decode_mm_kwargs_item), and site 5 adds an is_embed field to PlaceholderRangeInfo with from_placeholder_range/to_placeholder_range helpers so the sparse mask survives render-to-generate replay. Each site fix ships with a regression test. This packet groups the five sites because they share one entry point (/inference/v1/generate + /render) and one root cause; we are happy to split it into per-component advisories (for example, engine-fatal input-validation vs. cache-key binding vs. transport-schema integrity) if the vLLM team prefers.

Credit

Reported by: Patch the Planet (Trai