Skip to content

macOS wheels export bundled protobuf/abseil symbols → segfault on import alongside any other protobuf C++ (e.g. ortools) #115

Open

Description

@mmattice

Summary

On macOS, cel_expr_python crashes with SIGSEGV at import time if another
extension that ships its own protobuf C++ runtime is already loaded, or is
loaded afterwards. ortools is one such package. The crash happens in both
import orders. The same packages import fine together on Linux.

Reproduction

python3.13 -m venv v && . v/bin/activate
pip install cel-expr-python==0.1.3 ortools==9.15.6755
python -c "from ortools.sat.python import cp_model; from cel_expr_python import cel"   # exit 139
python -c "from cel_expr_python import cel; from ortools.sat.python import cp_model"   # exit 139

Each package imports fine on its own.

Environment

  • macOS 26.4.1, x86_64, CPython 3.13.16
  • cel-expr-python 0.1.3 (macosx_10_13_x86_64 wheel), ortools 9.15.6755, protobuf (Python) 6.33.6
  • Works on Linux (python:3.13-slim, amd64 and arm64) with the same pins.
  • The arm64 macOS wheel has the same structure (WEAK_DEFINES, weak google::protobuf exports). Coworkers on M2 silicon macs have run into the same issues.

Analysis

cel.cpython-313-darwin.so statically links protobuf C++ (33.x) and abseil
(lts_20260107), and it exports those symbols. The image is flagged
WEAK_DEFINES BINDS_TO_WEAK and exports about 2,281 weak definitions.
ortools ships protobuf 33.1 and abseil lts_20250814 as dylibs. On macOS,
dyld coalesces weak definitions across all loaded images, regardless of
RTLD_LOCAL or two-level namespace. The two images share 212 weak symbols,
including:

google::protobuf::FileDescriptorProto::_table_
google::protobuf::FeatureSet::_table_
google::protobuf::FeatureSetDefaults::_table_
google::protobuf::internal::MergeFromImpl<...>

Whichever image loads second binds to the first one's copies.
DYLD_PRINT_BINDINGS shows 125 binds from cel.so into ortools'
libprotobuf.33.1.0.dylib when ortools loads first. The second package's
static initializers then crash while registering descriptors:

  • ortools first: cel AddDescriptors → EncodedDescriptorDatabase::Add → MergeFromImpl<false> (ortools' copy) → TcParser::FastBS1 (bad read at 0xffffffe000000008)
  • cel first: ortools AddDescriptors → EncodedDescriptorDatabase::Add → ~FileDescriptorProto → RepeatedPtrFieldBase::DestroyProtos (bad read at 0x4)

sys.setdlopenflags(os.RTLD_LOCAL | os.RTLD_NOW) does not help, because
weak-def coalescing ignores those flags.

On Linux, the .so exports the same symbols (about 8.5k protobuf/abseil), but
RTLD_LOCAL keeps them private. LD_DEBUG=bindings shows only one
cross-binding, the STB_GNU_UNIQUE
google::protobuf::internal::cpp::IsTrackingEnabledVar()::is_tracking_enabled.

Suggested fix

Export only the module init symbol from the extension:

  • macOS: -Wl,-exported_symbol,_PyInit_cel (or an -exported_symbols_list)
  • Linux: a version script { global: PyInit_cel; local: *; };
  • Compile with -fvisibility=hidden -fvisibility-inlines-hidden as well.

This keeps the bundled protobuf and abseil private, so they can't collide
with any other protobuf in the process. Other packages that bundle protobuf
hit the same problem; see protocolbuffers/protobuf#2943.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions