[ptweezy]

igzip.decompress() and IGzipFile.read() with no size, which json.load(igzip.open(path)) calls, both end up in _GzipReader.readall(). On develop, that method decompresses into chunks of four times the input buffer size and joins them with bytes.join(). When the output doesn't fit in the first chunk, the join copies all of it, so readall() needs about twice the output in memory. For files, the chunks are 2 MiB, even for a small file.

With this PR, readall() decompresses into one bytes object when it knows how much compressed data is left. That's the case for bytes-like input, for streams that it has read to the end, and for regular files, where fstat() and tell() give the amount. When the buffer is full, _PyBytes_Resize() grows it to the size that the compression ratio so far predicts, plus an eighth. Each resize grows the buffer by at least a quarter and to at most eight times its size. The estimate only sizes the allocation, and readall() doesn't read the size field of the gzip trailer. A stream of unknown size that doesn't fit in the input buffer, such as a pipe, still joins chunks. In my Windows tests with regular files, doubling one buffer took up to 1.3 times as long as joining chunks.

The first buffer is four times the compressed bytes in the input buffer, and at least 16 KiB. Before choosing that size, readall() fills an empty input buffer, so a file that fits in the input buffer is read to the end without a call to fstat(). In my mingw-w64 build, checking fileno(), fstat(), and tell() added 20 to 35 µs to _GzipReader(f, 512 * 1024).readall() of a 64 KiB file. readall() fills the input buffer only outside a deflate block: at the end of a member, GzipReader_read_into_buffer() moves current_pos back over the bytes that are still in the bit buffer of ISA-L, so those bytes must stay in the input buffer. Without this check, test_readall_after_read_inside_deflate_block fails with "CRC check failed".

read() with a size, readinto(), readline(), and iteration don't use readall(). On that path, the only change is a counter of compressed bytes read. readall() now holds the object's lock for the whole call, where develop released it between chunks.

Results

I measured seven kinds of data (CPython's Lib/**/*.py files, three kinds of JSON event lines, FASTQ, CSV, and a web server log), compressed with igzip level 2, gzip -6, and zlib level 9 at eight sizes from 64 KiB to 128 MiB. That's 168 cases each for igzip.decompress() and igzip.open(path).read(), with ratios from 2.9 to 35 to 1. On each machine, I built both versions and timed them in thread CPU time, in fresh processes in a new random order for each case, in 9 rounds locally and 11 rounds on GitHub runners. Timing develop against itself the same way gives a noise floor of 2 to 6%.

The table shows how many times faster the first call in a process is with this PR: the median over the cases in each row, and the range.

Call	Linux container	Ubuntu runner	Windows runner	macOS runner
decompress(), ratio above 4.2 to 1, 1 to 128 MiB	1.63 (1.38-1.96)	1.38 (1.10-2.39)	1.15 (1.00-1.65)	1.09 (1.02-1.42)
decompress(), ratio below 4 to 1, 1 to 128 MiB	1.02 (0.97-1.04)	1.00 (0.98-1.05)	1.00 (0.98-1.02)	1.00 (0.95-1.07)
decompress(), 64 KiB	1.23 (0.96-1.29)	1.21 (0.97-1.31)	1.04 (0.96-1.21)	1.06 (0.89-1.27)
read(), 64 KiB	1.11 (1.08-1.17)	1.11 (1.08-1.14)	1.05 (0.95-1.09)	1.11 (0.97-1.35)
read(), 1 to 2 MiB	1.02 (0.94-1.07)	1.00 (0.92-1.01)	0.90 (0.72-1.00)	1.00 (0.96-1.05)
read(), 3 to 32 MiB	1.50 (1.28-1.97)	1.38 (1.14-2.80)	1.11 (0.93-1.34)	1.06 (0.99-1.53)
read(), 128 MiB	1.60 (1.36-1.99)	1.59 (1.26-3.13)	0.99 (0.97-1.00)	1.10 (1.02-1.37)

The Linux container is Docker on WSL2 on an AMD Ryzen 7 9800X3D (CPython 3.12.15, one pinned CPU), where a page fault costs about five times as much as on the Ubuntu runner. The runners use CPython 3.12, and the Windows runner uses an MSVC build; a mingw-w64 build on the Ryzen gives row medians within 0.07 of it. igzip.decompress() changes little below 4 to 1 because, for bytes input, the output then fits in the first chunk of develop, which is four times the compressed size.

In warm calls, which repeat the call in the same process, read() of a 64 KiB file is 1.6 times faster on the Ubuntu runner and 2.0 times faster in the Linux container, because the 2 MiB chunk of develop takes 18 page faults per call there. In first calls, json.load(igzip.open(path)) is 1.01 to 1.04 times faster (geometric mean per platform), because parsing takes most of the time. read(131072), readline(), and line iteration don't change.

With 32 MiB of output, read() needs 1.14 times the output in memory instead of 2.02 times, as traced by tracemalloc, and on every platform its peak RSS grows by about the output size instead of twice that. When later data compresses much worse than earlier data, the buffer overshoots, up to 3.9 times the output in my tests, but readall() touches only the part it writes, so the peak RSS doesn't grow more. On Windows, the rest still counts toward the commit charge, which I didn't measure.

Slower cases

• On Windows, read() of a file with 1 to 2 MiB of output takes 1.05 to 1.4 times as long in 35 of 42 first calls. Those are exactly the cases that need a resize, which copies the output on Windows, while develop writes these outputs into its 2 MiB first chunk. With 4 MiB of text, which needs two resizes, read() takes 1.07 times as long.
• On Windows, read() with 128 MiB of output is no faster, and its peak RSS grows by twice the output, as on develop. With the limit of eight times per resize, the buffer grows from 2 to 16 to 128 MiB and then, because the output fills it exactly, to 160 MiB, and that last resize copies 128 MiB. From the code, larger outputs need the same third resize. I didn't measure them.
• When the estimate is much too low, each resize grows the buffer by only a quarter. For 8 MiB of random data followed by 256 MiB of zeros or of one repeated line, readall() resizes 8 to 17 times. On Linux, where realloc() moves large blocks with mremap(), it's still 1.2 to 2.1 times faster than develop. On Windows, where each resize copies, it takes 1.8 to 2.3 times as long.
• On Linux, warm calls of igzip.decompress() on 64 KiB of log data compressed with gzip -6 take 1.6 to 2.1 times as long, with 16 page faults per call where develop takes none. First calls are 1.2 times faster.
• On PyPy 3.11 in the Linux container, first calls with 4 to 128 MiB of output are 1.0 to 1.5 times faster, but warm read() calls with 128 MiB take 1.2 to 1.3 times as long. Keeping the chunks on PyPy with #ifdef PYPY_VERSION removed that regression but also most of the gains, so I left it out.

Tests

The new file tests/test_readall.py has 50 tests. They cover bytes and files in five sizes, a stream without fileno(), io.BytesIO, the chunk path through a 4 KiB input buffer, members that compress very differently, 300 members of random sizes, readall() after read() calls of many sizes and inside a deflate block, files whose tell() fails or points past the end, and peak memory. On develop, only the four test_peak_memory tests fail.

The full test suite passes with CPython 3.12.15 on Linux, with CPython 3.14.8 and AddressSanitizer (leak detection off, no reports), with PyPy 3.11 (without the two compliance test files, which need CPython's test package), on Windows with mingw-w64 and MSVC builds, on macOS arm64, and on free-threaded CPython 3.14t. On 3.14t, a stress test with threads that call read() and readall() on one shared reader found no errors. With this PR, no readall() result had another thread's read in the middle of it; on develop, 298 did. flake8 and mypy pass as in tox.ini, and GCC with -Wall -Wextra gives the same warnings as on develop. This PR merges cleanly with each of #287, #288, and #289.

Checklist

• Pull request details were added to CHANGELOG.rst
• Documentation was updated (if needed)