Sockets opened outside profiling do not close properly inside profiling #342

Activity

1. altendky commented on Dec 25, 2021

   

   altendky

   on Dec 25, 2021

   CollaboratorAuthor

   More actions

   Mmm... multiprocessing is being used with the default start method of fork with which "All resources of the parent are inherited by the child process". Explicitly specifying spawn instead with multiprocessing.set_start_method("spawn") works. spawn is described in part with "unnecessary file descriptors and handles from the parent process will not be inherited".

   new commands that work

   cat > x.py << EOF
   import multiprocessing
   import socket
   
   import memory_profiler
   
   
   address = ("127.0.0.1", 33125)
   
   
   def main():
       multiprocessing.set_start_method("spawn")
   
       sock = socket.socket()
       sock.bind(address)
   
       profiled(sock)
   
       sock3 = socket.socket()
       sock3.bind(address)
   
       memory_profiler.memory_usage((profiled, [sock3]))
       print("really made it")
   
   
   def profiled(sock):
       sock.close()
   
       sock2 = socket.socket()
       sock2.bind(address)
       sock2.close()
   
   
   # auaughhghgh  multiprocessing!
   if __name__ == "__main__":
       main()
   EOF
   cat x.py
   python3.9 -m venv venv
   venv/bin/python -m pip install --upgrade pip setuptools wheel
   venv/bin/pip install memory-profiler==0.60.0 psutil==5.8.0
   venv/bin/python x.py
   venv/bin/python --version --version
   venv/bin/pip freeze
   uname -a
   lsb_release -a
   

   new working terminal session

   $ cat > x.py << EOF
   > import multiprocessing
   > import socket
   > 
   > import memory_profiler
   > 
   > 
   > address = ("127.0.0.1", 33125)
   > 
   > 
   > def main():
   >     multiprocessing.set_start_method("spawn")
   > 
   >     sock = socket.socket()
   >     sock.bind(address)
   > 
   >     profiled(sock)
   > 
   >     sock3 = socket.socket()
   >     sock3.bind(address)
   > 
   >     memory_profiler.memory_usage((profiled, [sock3]))
   >     print("really made it")
   > 
   > 
   > def profiled(sock):
   >     sock.close()
   > 
   >     sock2 = socket.socket()
   >     sock2.bind(address)
   >     sock2.close()
   > 
   > 
   > # auaughhghgh  multiprocessing!
   > if __name__ == "__main__":
   >     main()
   > EOF

   $ cat x.py
   import multiprocessing
   import socket
   
   import memory_profiler
   
   
   address = ("127.0.0.1", 33125)
   
   
   def main():
       multiprocessing.set_start_method("spawn")
   
       sock = socket.socket()
       sock.bind(address)
   
       profiled(sock)
   
       sock3 = socket.socket()
       sock3.bind(address)
   
       memory_profiler.memory_usage((profiled, [sock3]))
       print("really made it")
   
   
   def profiled(sock):
       sock.close()
   
       sock2 = socket.socket()
       sock2.bind(address)
       sock2.close()
   
   
   # auaughhghgh  multiprocessing!
   if __name__ == "__main__":
       main()

   $ python3.9 -m venv venv

   $ venv/bin/python -m pip install --upgrade pip setuptools wheel
   Requirement already satisfied: pip in ./venv/lib/python3.9/site-packages (21.1.1)
   Collecting pip
     Using cached pip-21.3.1-py3-none-any.whl (1.7 MB)
   Requirement already satisfied: setuptools in ./venv/lib/python3.9/site-packages (56.0.0)
   Collecting setuptools
     Using cached setuptools-60.1.0-py3-none-any.whl (952 kB)
   Collecting wheel
     Using cached wheel-0.37.1-py2.py3-none-any.whl (35 kB)
   Installing collected packages: wheel, setuptools, pip
     Attempting uninstall: setuptools
       Found existing installation: setuptools 56.0.0
       Uninstalling setuptools-56.0.0:
         Successfully uninstalled setuptools-56.0.0
     Attempting uninstall: pip
       Found existing installation: pip 21.1.1
       Uninstalling pip-21.1.1:
         Successfully uninstalled pip-21.1.1
   Successfully installed pip-21.3.1 setuptools-60.1.0 wheel-0.37.1

   $ venv/bin/pip install memory-profiler==0.60.0 psutil==5.8.0
   Collecting memory-profiler==0.60.0
     Using cached memory_profiler-0.60.0-py3-none-any.whl
   Collecting psutil==5.8.0
     Using cached psutil-5.8.0-cp39-cp39-manylinux2010_x86_64.whl (293 kB)
   Installing collected packages: psutil, memory-profiler
   Successfully installed memory-profiler-0.60.0 psutil-5.8.0

   $ venv/bin/python x.py
   really made it

   $ venv/bin/python --version --version
   Python 3.9.5 (default, Jun  3 2021, 15:18:23) 
   [GCC 9.3.0]

   $ venv/bin/pip freeze
   memory-profiler==0.60.0
   psutil==5.8.0

   $ uname -a
   Linux p1 5.4.0-91-generic #102-Ubuntu SMP Fri Nov 5 16:31:28 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

   $ lsb_release -a
   No LSB modules are available.
   Distributor ID: Ubuntu
   Description:    Ubuntu 20.04.3 LTS
   Release:        20.04
   Codename:       focal

   Ok, so with the cause identified, what next? I'm not exactly sure. Probably some documentation commenting on this with references to each of multiprocessing, spawn, fork, inherit-ed/ing, file descriptors, etc so it is findable by searching. I hesitate to mess with the multiprocessing global state and auto-configure this since code using memory-profiler may choose something else. If we do set the start method then it should only happen if not otherwise configured by the 'outer' code. I'll think it over and see what I can come up with.
2. altendky commented on Dec 29, 2021

   

   altendky

   on Dec 29, 2021

   CollaboratorAuthor

   More actions

   Are multiprocessing features really being used? Or could we just use a regular subprocess instead?