Skip to content

Fix int overflow in PCX image buffer sizing, now throws AllocationRequestException#552

Merged
garydgregory merged 3 commits into
apache:masterfrom
alhudz:fix-pcx-size-overflow
Jun 23, 2026
Merged

Fix int overflow in PCX image buffer sizing, now throws AllocationRequestException#552
garydgregory merged 3 commits into
apache:masterfrom
alhudz:fix-pcx-size-overflow

Conversation

@alhudz

@alhudz alhudz commented Jun 5, 2026

Copy link
Copy Markdown
Contributor

Reading the PCX decoder I noticed getBufferedImage sizes the pixel buffer with int multiplies of xSize/ySize, which both come from uint16 header fields and can reach 65536. For 24/32-bit and 8-bit/3-plane images rowLength * ySize, xSize * ySize and ySize * bytesPerImageRow overflow int and wrap small, so the value slips past the Allocator 1 GB guard and an undersized buffer feeds an out-of-bounds arraycopy. Computing in long lets the existing Allocator.byteArray(long) overload reject the real size instead.

@garydgregory garydgregory left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@alhudz
There is no test for this PR.

@alhudz

alhudz commented Jun 22, 2026

Copy link
Copy Markdown
Contributor Author

Added one in PcxImageParserTest. It feeds a crafted 128-byte header with bitsPerPixel=24, nPlanes=1 and xMax=yMax=65535 (so xSize=ySize=65536), which makes rowLength * ySize (196608 * 65536) overflow int and wrap small.

  • expected: AllocationRequestException from the 1 GB Allocator guard
  • before the fix: the wrapped size allocates an undersized buffer and you get an ArrayIndexOutOfBoundsException from the arraycopy
  • after: the size is computed in long so the guard sees the real ~12 GB request and rejects it

Verified the test fails on master and passes with the fix.

@garydgregory

Copy link
Copy Markdown
Member

@alhudz
As stated in the PR template, run 'mvn' by itself, and fix issues before you push.

@alhudz

alhudz commented Jun 23, 2026

Copy link
Copy Markdown
Contributor Author

Good catch. Running mvn flagged a checkstyle LineLength violation in the test javadoc (line was 161 chars, limit 160). I've wrapped it and the default goal is green now: clean verify apache-rat:check checkstyle:check spotbugs:check pmd:check all pass with the new test included.

@garydgregory garydgregory changed the title fix int overflow in pcx image buffer sizing Fix int overflow in pcx image buffer sizing Jun 23, 2026
@garydgregory garydgregory merged commit e92fa1d into apache:master Jun 23, 2026
17 checks passed
@garydgregory garydgregory changed the title Fix int overflow in pcx image buffer sizing Fix int overflow in PCX image buffer sizing, now throws AllocationRequestException Jun 23, 2026
garydgregory added a commit that referenced this pull request Jun 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants