Skip to content

Release GIL during image encoding and map loading - #301

Open
JabberWocky-22 wants to merge 2 commits into
mapnik:masterfrom
JabberWocky-22:fix/image-encoding-gil
Open

JabberWocky-22 wants to merge 2 commits into
mapnik:masterfrom
JabberWocky-22:fix/image-encoding-gil

Conversation

@JabberWocky-22

@JabberWocky-22 JabberWocky-22 commented Oct 17, 2026 •

Copy link
Copy Markdown

Fixes #300.

Formatted Image.to_string() and ImageView.to_string() encoding currently hold the GIL, so encoding in a Python worker thread still blocks other Python threads.

Release the GIL only around the native save_to_string() calls in all four formatted overloads, including explicit-palette variants. The GIL is reacquired before constructing Python bytes, including on the exception path. Encoding algorithms and parameters are unchanged.

Add a parametrized regression test covering Image/ImageView with and without a palette. It checks that another Python thread can run during encoding and that decoded pixels are preserved, without latency thresholds. Also add a changelog entry.

Validation

Built the native Mapnik v4.3.2 core and original/patched Python bindings in an isolated Docker environment:

  • Original bindings: all four new GIL regression cases fail as expected.
  • Patched bindings: 68 passed across image_test.py, png_encoding_test.py, palette_test.py, image_tiff_test.py, and webp_encoding_test.py.
  • Repeated the four new regression cases 20 times: 80 passed.
  • Compared original and patched output using identical native dependencies: 11 encoded outputs were byte-for-byte identical, covering supported Image/ImageView combinations of PNG, PNG256, JPEG, WebP, TIFF, and explicit-palette PNG.

No C++ coverage report was collected.

Additional fix: map loading

load_map() and load_map_from_string() have the same GIL retention problem. Datasource initialization during loading can perform database I/O, so loading in a worker thread may still block other Python threads while waiting for a database.

Add py::call_guard<py::gil_scoped_release>() to both bindings. Argument conversion happens before the GIL is released, and the GIL is reacquired when the native call returns or throws. The existing image-encoding commit is retained; this fix is a separate additional commit.

Add one parametrized regression test for file and string loading. It checks thread progress during loading and verifies that all 2,000 layers are loaded, without latency thresholds or external database dependencies. Add a changelog entry.

Additional validation in an isolated Docker container:

  • Original v4.3.2 bindings: both new map-loading GIL regression cases fail as expected.
  • Compiled the two modified binding definitions against Mapnik v4.3.2, reusing the installed core and remaining Python bindings: 5 passed across load_map_test.py, including the two new cases.

This additional validation did not rebuild the complete Python extension.


Implementation, tests, and PR text generated with Codex.

JabberWocky-22 and others added 2 commits October 17, 2026 15:21
Release the GIL around load_map and load_map_from_string using a pybind11 call guard. Add a parametrized regression test for both loading paths and document the fix in the changelog.

Generated with Codex.
@JabberWocky-22 JabberWocky-22 changed the title Release GIL during Image and ImageView encoding Release GIL during image encoding and map loading Oct 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Image encoding and map loading hold the Python GIL

1 participant