RabbitMQ Java client has frame-level OOM: Math.min(maxInboundMessageBodySize, 0) defeats frame size enforcement

Vulnerability

In AMQConnection.java (line 435-436), after Connection.Tune negotiation, the frame-max limit is set via:

_frameHandler.setFrameMax(
    Math.min(this.maxInboundMessageBodySize, frameMax));

When frameMax = 0 (meaning "unlimited" per AMQP spec), Math.min(67108864, 0) = 0. This value is then passed to Utils.framePayloadLimit(0) which returns Integer.MAX_VALUE (line 77-79 of Utils.java):

static int framePayloadLimit(int frameMax) {
    if (frameMax <= 0) {
      return Integer.MAX_VALUE;
    }
    // ...
}

This completely defeats the maxInboundMessageBodySize protection (default 64MB) at the frame level.

Attack Scenario

A malicious AMQP server (or MITM) sends Connection.Tune with frameMax=0:

1. Client defaults: requestedFrameMax = 0 (ConnectionFactory.DEFAULT_FRAME_MAX, line 82)
2. negotiatedMaxValue(0, 0) = Math.max(0, 0) = 0 (line 673-676)
3. Math.min(maxInboundMessageBodySize, 0) = 0 — 64MB cap defeated
4. framePayloadLimit(0) = Integer.MAX_VALUE — no frame size enforcement
5. Attacker sends a single frame with frameSize = 0x1FFFFFFF (~500MB)
6. Frame.readFrom() (line 135) executes new byte[frameSize] — OOM crash

The frame does not need to be a body frame — method frames, header frames, or heartbeat frames with a crafted size field all trigger the allocation before any content-level check fires.

Root Cause

The AMQP spec uses frameMax=0 to mean "unlimited", but Math.min treats it as the integer value zero. The intent of line 435-436 was to take the smaller of the two limits, but when one limit uses 0-means-unlimited semantics, Math.min always selects the zero, disabling the other limit.

Impact

• Default configuration is vulnerable: Both requestedFrameMax (client) and legitimate servers' frameMax in Tune may be 0
• Single-frame OOM: One malicious frame triggers up to ~2GB allocation (Integer.MAX_VALUE bytes)