vLLM: Forged multimodal UUID + P0/P1 processor-cache eviction drift: unauthenticated remote crash of the whole vLLM engine

Details

1. The client-controlled uuid becomes the cache hash verbatim. vllm/entrypoints/chat_utils.py:1677 / :1884-1886 read part["uuid"] per content part and pass it through as mm_uuids. In vllm/multimodal/processing/inputs.py:62-63, when no hf_processor_mm_kwargs are present (the default for chat requests), the uuid string is the item hash — no content binding at all:

# vllm/multimodal/processing/inputs.py:62-63
else:
    hashes.append(uuid_item)        # client string used verbatim as mm_hash

2. P0 insertion happens before request validation. In the serving path, render_chat_request (vllm/entrypoints/openai/chat_completion/serving.py:275) runs full mm processing — mm_processor.apply (vllm/renderers/base.py:759) → _cached_apply_hf_processor → _merge_mm_kwargs → cache.get_and_update_item (vllm/multimodal/processing/processor.py:1338), which inserts the processed item into the P0 sender cache (default mm_processor_cache_type="lru", 4 GiB). Only afterwards does get_max_tokens (vllm/entrypoints/serve/utils/api_utils.py:183-186) reject over-long prompts with 400. A request can therefore write to P0 while never reaching the engine — the primitive that desynchronizes P0 from P1. (Public issue #43941 documented this same asymmetry for the opposite drift direction, which was later handled via MultiModalCacheMissError; the P0-miss/P1-hit direction below was never addressed.)

3. P1 substitutes cached data on hash hit with no consistency check. In the EngineCore input thread, preprocess_add_request (vllm/v1/engine/core.py:997-1000) calls mm_receiver_cache.get_and_update_features, which only replaces feature.data (vllm/multimodal/cache.py:673-690) and keeps the request's own mm_position (placeholder layout). On a cache hit the receiver cache returns the stale item wholesale:

# vllm/multimodal/cache.py:739-740  (MultiModalReceiverCache.get_and_update_item)
if (cached_item := self._cache.get(mm_hash)) is not None:
    return cached_item              # fresh data silently discarded; no size/modality check

Only the opposite drift (P0 hit / P1 miss, data=None) is handled, via the retryable MultiModalCacheMissError. Nothing validates that the cached item's modality, shape, or token count matches the current request's placeholders.

4. The mismatch is fatal to the engine. The scheduler plans encoder work from the request's (fresh) placeholder count N_B, while the worker gathers embeddings from the stale N_A-row tensor (encoder-cache hit under the same forged hash, or re-encoding the stale data — both yield N_A rows). _merge_multimodal_embeddings then raises:

# vllm/model_executor/models/utils.py:703-714
inputs_embeds[is_multimodal] = mm_embeds_flat.to(dtype=input_dtype)   # RuntimeError
...
raise ValueError(
    f"Attempted to assign {expr} = {num_actual_tokens} "
    f"multimodal tokens to {num_expected_tokens} placeholders") from e

This exception is raised inside the worker's input preparation with no per-request containment, propagates to the EngineCore busy loop's top-level handler (vllm/v1/engine/core.py:1372), and triggers _send_engine_dead(). The API server's output_handler receives EngineDeadError (vllm/v1/engine/async_llm.py:819) and the whole process tree shuts down. (Issue #55546 independently reported this sink — "one bad multimodal item kills the entire V1 engine tree" — from an organically occurring stale cache entry; this report provides the missing deterministic, remote, single-attacker trigger.)

Cost of the attack. With the default mm_processor_cache_gb=4, one max-size Qwen2.5-VL image (12845056 pixels) caches as ~77 MB, so ~56 flood requests suffice; we used 70. Flood requests are CPU-only image decodes (~0.3 s each here) and can be issued sequentially or concurrently; the total wall-clock cost of the kill was under one minute. Models with larger max_pixels fill the cache in even fewer requests. Note that issuing the flood with successful requests does not work — they would populate P1 as well; rejection after P0 insertion is essential, and the over-long-prompt 400 is one always-available way to get it (no size limits depend on non-default flags).

Impact

Any unauthenticated client that can reach a multimodal vLLM endpoint (/v1/chat/completions, /v1/responses, etc. — all accept content-part uuid) can deterministically crash the entire serving instance: the EngineCore process dies, every in-flight request fails, and the API server exits; the deployment stays down until restarted. In data-parallel or multi-API-server deployments the blast radius extends to the whole engine group (cf. the DP8 outage in #55546). The attack requires no special privileges, no non-default server flags, no race conditions, and costs under a minute of sequential requests at default cache sizes.

This is distinct from the documented "Multimodal Media UUID Security" guidance in docs/usage/security.md, which covers only integrity/confidentiality consequences of uuid collisions (a caller silently receiving another caller's media). The guidance does not mention availability, and following it does not help: the attacker here deliberately collides with their own uuid, and the fatal element is the missing consistency check on the P1 hit path plus the unsynchronized eviction of the two caches.

Suggested remediation (any one of these breaks the kill chain):

1. On a P1 receiver-cache hit, validate the cached item against the request's mm_position (modality and expected embedding count); on mismatch, treat it as a miss (MultiModalCacheMissError) or reject the request — never substitute silently.
2. Make the feature/placeholder count mismatch in _merge_multimodal_embeddings a per-request error instead of a fatal engine exception (also requested by #55546).
3. Bind server-side cache identity to content even when a uuid is provided (e.g. hash(content) + uuid), eliminating forged-hash cache hits entirely.
4. Synchronize P0/P1 evictions (the engine already pushes invalidate to P0 on cache-miss; eviction-direction sync would close the remaining drift).

A public fix is available here: https://github.com/vllm-project/vllm/pull/57833