Netty: HTTP/2 decompression leaks ByteBuf reference count when the decompressor channel is already closed (Direct memory leak / OOM DoS)

Suggested fix: track whether writeInbound succeeded and roll back the extra retain() only when the data never entered the pipeline:

boolean writeSucceeded = false;
try {
    decompressor.writeInbound(data.retain());
    writeSucceeded = true;            // pipeline now owns the release
    if (endOfStream) {
        decompressor.finish();
    }
    return 0;
} catch (Throwable t) {
    if (!writeSucceeded) {
        data.release();               // roll back the extra retain(); data never entered pipeline
    }
    if (t instanceof Http2Exception) {
        throw (Http2Exception) t;
    }
    throw streamError(stream.id(), INTERNAL_ERROR, t, ...);
}

PoC

Reproduced against the official, unmodified netty-codec-http2-4.2.15.Final.jar from Maven Central, using real netty classes and measuring ByteBuf.refCnt() directly (the leaking logic is not mocked).

Reproduction steps:

1. Download the official artifacts and their dependencies from Maven Central (version 4.2.15.Final): netty-common, netty-buffer, netty-transport, netty-resolver, netty-handler, netty-codec-base, netty-codec, netty-codec-http, netty-codec-http2, netty-codec-compression.
2. Build a real Http2Decompressor wrapping a real gzip decoder EmbeddedChannel (ZlibCodecFactory.newZlibDecoder(ZlibWrapper.GZIP)).
3. Close the internal decompressor channel (equivalent to the end state of cleanup() / finishAndReleaseAll()).
4. Encode a real gzip DATA payload with ZlibCodecFactory.newZlibEncoder(GZIP) (refCnt = 1).
5. Call decompress(...) on the closed channel.
6. Observe: writeInbound(...) throws ClosedChannelException at its ensureOpen() entry (EmbeddedChannel.java:360), reached from DelegatingDecompressorFrameListener.java:433; data.refCnt() is now 2.
7. Release once as the frame reader would; refCnt stays at 1 (release() returns false) → leaked.

Observed reference-count trace:

gzipData initial refCnt = 1
decompress -> data.retain()        -> refCnt = 2   (retain applied, never rolled back)
caller releases once               -> refCnt = 1   (release() returns false; not deallocated)
=> buffer never reaches 0 -> direct memory leaked

Observed exception stack (confirms the leak point):

java.nio.channels.ClosedChannelException
    at io.netty.channel.embedded.EmbeddedChannel.checkOpen(EmbeddedChannel.java:959)
    at io.netty.channel.embedded.EmbeddedChannel.ensureOpen(EmbeddedChannel.java:979)
    at io.netty.channel.embedded.EmbeddedChannel.writeInbound(EmbeddedChannel.java:360)
    at io.netty.handler.codec.http2.DelegatingDecompressorFrameListener$Http2Decompressor
            .decompress(DelegatingDecompressorFrameListener.java:433)

Two notes on the harness (they do not affect the leak mechanism):

• The internal channel is closed directly via close() rather than through cleanup(). The end state is identical (channel closed → writeInbound throws at ensureOpen()); the bug depends on "channel closed → retain not rolled back", not on how the channel was closed.
• In the isolated harness the rethrown StreamException's root cause shows as NullPointerException because the harness does not initialise an Http2LocalFlowController (a secondary exception reported during channel close). The leak is already sealed at the ClosedChannelException thrown by writeInbound's ensureOpen() (line 360); in a real server with the flow controller initialised, the triggering exception is the ClosedChannelException itself.

A complete self-contained PoC (Verify02DecompressLeak.java, ~150 lines, no test framework) plus the exact javac / java commands can be attached on request.

Impact

• Vulnerability type: uncontrolled resource consumption / memory leak (CWE-401), leading to denial of service. Each crafted DATA frame leaks one (typically direct/off-heap) ByteBuf.
• Who is impacted: any server (or client) that enables HTTP/2 content decompression by installing DelegatingDecompressorFrameListener in its HTTP/2 pipeline.
• Attacker requirements: remote, unauthenticated. The attacker only needs to send HTTP/2 DATA frames for a stream whose decompressor has been cleaned up (e.g. continue sending DATA after END_STREAM). No special server configuration beyond decompression being enabled.
• Result: sustained triggering over a long-lived connection exhausts direct memory and crashes the JVM with OutOfMemoryError.