Repository navigation
multiprocessing.managers proxy finalizer can deadlock a manager server running in the same process #159986
Description
Bug report
When a multiprocessing.managers proxy is garbage collected, its finalizer, BaseProxy._decref, sends a decref request to the manager's server over a new connection and blocks until the server replies. The finalizer runs on whichever thread frees the proxy. For a proxy that is only reachable from a reference cycle, that is whichever thread happens to trigger the cyclic garbage collector. To serve the request, the server's accepter thread has to accept the connection and start a request thread.
This can deadlock when the server runs in the same process as the proxy, for example via BaseManager.get_server().serve_forever() on a background thread. If the GC is triggered on the accepter thread itself, the finalizer waits for a request thread that only the accepter could start, and the server never accepts another connection. LLMs tell me the deadlock might happen also if the GC runs on a request thread inside Server.create.
Observed in apache/beam#40673. It may be a corner case, since the proxy and the server usually live in different processes. Still, it may be worth making _decref non-blocking, perhaps by handing the decref to a background thread.
The repro below deadlocks within a few iterations on 3.10-3.14; gc.set_threshold(10) makes the race easy to hit. In the faulthandler dump, the accepter thread is blocked in answer_challenge, called from BaseProxy._decref through the proxy's finalizer while the accepter constructs a Thread.
import faulthandler
import gc
import threading
from multiprocessing.managers import BaseManager
class Counter:
def __init__(self):
self.n = 0
def incr(self):
self.n += 1
return self.n
class MyManager(BaseManager):
pass
MyManager.register('Counter', Counter)
if __name__ == '__main__':
# Serve the manager from a thread of this process.
server = MyManager(address=('127.0.0.1', 0), authkey=b'secret').get_server()
threading.Thread(target=server.serve_forever, daemon=True).start()
manager = MyManager(address=server.address, authkey=b'secret')
manager.connect()
# Run the cyclic GC often, on whichever thread happens to allocate.
gc.set_threshold(10)
def work():
for _ in range(1000):
proxy = manager.Counter()
proxy.incr()
cycle = [proxy]
cycle.append(cycle) # Only the cyclic GC can free the proxy.
t = threading.Thread(target=work, daemon=True)
t.start()
t.join(timeout=30)
if t.is_alive():
print('DEADLOCKED')
faulthandler.dump_traceback(all_threads=True)
else:
print('ok')Output
DEADLOCKED
Thread 0x00007f0d665266c0 [MainProcess|Thr] (most recent call first):
File "/usr/.../python3.14/multiprocessing/connection.py", line 412 in _recv
File "/usr/.../python3.14/multiprocessing/connection.py", line 447 in _recv_bytes
File "/usr/.../python3.14/multiprocessing/connection.py", line 256 in recv
File "/usr/.../python3.14/multiprocessing/managers.py", line 257 in serve_client
File "/usr/.../python3.14/multiprocessing/managers.py", line 425 in accept_connection
File "/usr/.../python3.14/multiprocessing/managers.py", line 213 in _handle_request
File "/usr/.../python3.14/multiprocessing/managers.py", line 235 in handle_request
File "/usr/.../python3.14/threading.py", line 1023 in run
File "/usr/.../python3.14/threading.py", line 1081 in _bootstrap_inner
File "/usr/.../python3.14/threading.py", line 1043 in _bootstrap
Thread 0x00007f0d65d256c0 [Thread-4 (work)] (most recent call first):
File "/usr/.../python3.14/multiprocessing/connection.py", line 412 in _recv
File "/usr/.../python3.14/multiprocessing/connection.py", line 447 in _recv_bytes
File "/usr/.../python3.14/multiprocessing/connection.py", line 222 in recv_bytes
File "/usr/.../python3.14/multiprocessing/connection.py", line 970 in answer_challenge
File "/usr/.../python3.14/multiprocessing/connection.py", line 542 in Client
File "/usr/.../python3.14/multiprocessing/managers.py", line 863 in _incref
File "/usr/.../python3.14/multiprocessing/managers.py", line 806 in __init__
File "/usr/.../python3.14/multiprocessing/managers.py", line 1007 in AutoProxy
File "/usr/.../python3.14/multiprocessing/managers.py", line 731 in temp
File "<python-input-0>", line 37 in work
File "/usr/.../python3.14/threading.py", line 1023 in run
File "/usr/.../python3.14/threading.py", line 1081 in _bootstrap_inner
File "/usr/.../python3.14/threading.py", line 1043 in _bootstrap
Thread 0x00007f0d66d276c0 [Thread-2 (accep] (most recent call first):
File "/usr/.../python3.14/multiprocessing/connection.py", line 412 in _recv
File "/usr/.../python3.14/multiprocessing/connection.py", line 447 in _recv_bytes
File "/usr/.../python3.14/multiprocessing/connection.py", line 222 in recv_bytes
File "/usr/.../python3.14/multiprocessing/connection.py", line 970 in answer_challenge
File "/usr/.../python3.14/multiprocessing/connection.py", line 542 in Client
File "/usr/.../python3.14/multiprocessing/managers.py", line 887 in _decref
File "/usr/.../python3.14/multiprocessing/util.py", line 291 in __call__
File "/usr/.../python3.14/threading.py", line 1307 in _make_invoke_excepthook
File "/usr/.../python3.14/threading.py", line 943 in __init__
File "/usr/.../python3.14/multiprocessing/managers.py", line 196 in accepter
File "/usr/.../python3.14/threading.py", line 1023 in run
File "/usr/.../python3.14/threading.py", line 1081 in _bootstrap_inner
File "/usr/.../python3.14/threading.py", line 1043 in _bootstrap
Thread 0x00007f0d675286c0 [Thread-1 (serve] (most recent call first):
File "/usr/.../python3.14/threading.py", line 373 in wait
File "/usr/.../python3.14/threading.py", line 669 in wait
File "/usr/.../python3.14/multiprocessing/managers.py", line 180 in serve_forever
File "/usr/.../python3.14/threading.py", line 1023 in run
File "/usr/.../python3.14/threading.py", line 1081 in _bootstrap_inner
File "/usr/.../python3.14/threading.py", line 1043 in _bootstrap
Current thread 0x00007f0d69528f00 [python] (most recent call first):
Garbage-collecting
File "<python-input-0>", line 47 in <module>
File "/usr/.../python3.14/_pyrepl/console.py", line 181 in runcode
File "/usr/.../python3.14/_pyrepl/console.py", line 226 in runsource
File "/usr/.../python3.14/code.py", line 324 in push
File "/usr/.../python3.14/_pyrepl/simple_interact.py", line 151 in run_multiline_interactive_console
File "/usr/.../python3.14/_pyrepl/main.py", line 58 in interactive_console
CPython versions tested on:
3.14
Operating systems tested on:
Linux
Activity
pythongh-159986: Defer local manager proxy finalization