Repository navigation
When making client calls from a sync env (with asyncio.run()) the loop gets closed after the first request, causing subsequent ones to fail #366
Copy link
Copy link
Closed
Labels
bugSomething isn't workingSomething isn't working
Description
Whenever I make requests via the GraphServiceClient from a sync environment using asyncio.run() or asgiref.sync.async_to_sync() the first call succeeds, however, all subsequent calls fail with an error:
Traceback (most recent call last):
File "/app/some_path/some_file.py", line 15, in get_drives
drives = asyncio.run(client.drives.get())
File "/usr/local/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
return future.result()
File "/usr/local/lib/python3.11/site-packages/msgraph/generated/drives/drive_request_builder.py", line 66, in get
return await self.request_adapter.send_async(request_info, DriveItemCollectionResponse, error_mapping)
File "/usr/local/lib/python3.11/site-packages/kiota_http/httpx_request_adapter.py", line 166, in send_async
response = await self.get_http_response_message(request_info, parent_span)
File "/usr/local/lib/python3.11/site-packages/kiota_http/httpx_request_adapter.py", line 508, in get_http_response_message
resp = await self._http_client.send(request)
File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1617, in send
response = await self._send_handling_auth(
File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1645, in _send_handling_auth
response = await self._send_handling_redirects(
File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1682, in _send_handling_redirects
response = await self._send_single_request(request)
File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1719, in _send_single_request
response = await transport.handle_async_request(request)
File "/usr/local/lib/python3.11/site-packages/msgraph_core/middleware/async_graph_transport.py", line 21, in handle_async_request
response = await self.pipeline.send(request)
File "/usr/local/lib/python3.11/site-packages/kiota_http/middleware/middleware.py", line 38, in send
return await self._first_middleware.send(request, self._transport)
File "/usr/local/lib/python3.11/site-packages/kiota_http/middleware/redirect_handler.py", line 77, in send
response = await super().send(request, transport)
File "/usr/local/lib/python3.11/site-packages/kiota_http/middleware/middleware.py", line 65, in send
return await self.next.send(request, transport)
File "/usr/local/lib/python3.11/site-packages/kiota_http/middleware/retry_handler.py", line 83, in send
response = await super().send(request, transport)
File "/usr/local/lib/python3.11/site-packages/kiota_http/middleware/middleware.py", line 65, in send
return await self.next.send(request, transport)
File "/usr/local/lib/python3.11/site-packages/kiota_http/middleware/parameters_name_decoding_handler.py", line 55, in send
response = await super().send(request, transport)
File "/usr/local/lib/python3.11/site-packages/kiota_http/middleware/middleware.py", line 65, in send
return await self.next.send(request, transport)
File "/usr/local/lib/python3.11/site-packages/kiota_http/middleware/url_replace_handler.py", line 44, in send
response = await super().send(request, transport)
File "/usr/local/lib/python3.11/site-packages/kiota_http/middleware/middleware.py", line 65, in send
return await self.next.send(request, transport)
File "/usr/local/lib/python3.11/site-packages/kiota_http/middleware/user_agent_handler.py", line 29, in send
return await super().send(request, transport)
File "/usr/local/lib/python3.11/site-packages/kiota_http/middleware/middleware.py", line 65, in send
return await self.next.send(request, transport)
File "/usr/local/lib/python3.11/site-packages/msgraph_core/middleware/telemetry.py", line 48, in send
response = await super().send(request, transport)
File "/usr/local/lib/python3.11/site-packages/kiota_http/middleware/middleware.py", line 62, in send
response = await transport.handle_async_request(request)
File "/usr/local/lib/python3.11/site-packages/msgraph_core/middleware/async_graph_transport.py", line 24, in handle_async_request
response = await self.transport.handle_async_request(request)
File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
resp = await self._pool.handle_async_request(req)
File "/usr/local/lib/python3.11/site-packages/httpcore/_async/connection_pool.py", line 226, in handle_async_request
await self._close_expired_connections()
File "/usr/local/lib/python3.11/site-packages/httpcore/_async/connection_pool.py", line 194, in _close_expired_connections
await connection.aclose()
File "/usr/local/lib/python3.11/site-packages/httpcore/_async/connection.py", line 166, in aclose
await self._connection.aclose()
File "/usr/local/lib/python3.11/site-packages/httpcore/_async/http11.py", line 241, in aclose
await self._network_stream.aclose()
File "/usr/local/lib/python3.11/site-packages/httpcore/_backends/anyio.py", line 54, in aclose
await self._stream.aclose()
File "/usr/local/lib/python3.11/site-packages/anyio/streams/tls.py", line 195, in aclose
await self.transport_stream.aclose()
File "/usr/local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 1161, in aclose
self._transport.close()
File "/usr/local/lib/python3.11/asyncio/selector_events.py", line 860, in close
self._loop.call_soon(self._call_connection_lost, None)
File "/usr/local/lib/python3.11/asyncio/base_events.py", line 761, in call_soon
self._check_closed()
File "/usr/local/lib/python3.11/asyncio/base_events.py", line 519, in _check_closed
raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closedThe simplest example that reproduces this is:
import asyncio
from azure.identity import ClientSecretCredential
from msgraph import GraphServiceClient
credential = ClientSecretCredential(
'TENANT_ID',
'CLIENT_ID',
'CLIENT_SECRET')
scopes = ["https://graph.microsoft.com/.default"]
client = GraphServiceClient(credential, scopes)
drives_1 = asyncio.run(client.drives.get())
drives_2 = asyncio.run(client.drives.get())I use the GraphServiceClient inside a task run in a background worker (Django Q) alongside my Django app. So the first time the background task runs it succeeds, while the second time it fails.
As a workaround this does the trick:
def get_loop():
try:
loop = asyncio.get_event_loop()
except RuntimeError:
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
return loop
# inside the background task
drives = get_loop().run_until_complete(client.drives.get())Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't working
Activity
stvpwrs commentedon Nov 14, 2023
Hey Andriy Berezhanskyy (@aberezh) I have encountered this issue too. For me, the issue was because I was running on Windows and I needed to add the following to my script:
That fixed the issue for me.
Good luck!
preslavnolimit commentedon Nov 14, 2023
I am having the same issue on Mac with Apple Silicon M2.
I manage to duck tape it similar to the example, but the exact example gives that "Future is not in same loop" on the second call. So I basically did that
This works for handling the "Event loop is closed problem", but sometimes I get http error from httpx timeout not sure if I am making it or the graph api is responsible.
The strange thing about my solution derived from the author of the issue #366 (comment)
is that asyncio.run actually does the same thing, not any different and it should work only with asyncio.run if look into the source code of the function "run".
steph409 commentedon Nov 15, 2023
I have the same problem running on Linux.
pclausen1 commentedon Nov 28, 2023
I had the same problem running in python 3.10 in Linux container. Was able to use similar workaround as others
I see though that get_event_loop() is deprecated in python 3.12, so perhaps need to replace with something similar to Andriy Berezhanskyy (@aberezh) solution, but with get_running_loop() instead
advsysal commentedon Dec 13, 2023
I have found a straightforward solution that seems to work across multiple hosting platforms (windows, databricks notebooks). It's similar to some of the above solutions, but a 1 liner...
instead of:
asyncio.run(my_graph_function())
use:
asyncio.get_event_loop().run_until_complete(my_graph_function())
erpola commentedon Dec 26, 2023
calling the
asyncio.runasget_loop().run_until_complete()works greats for this issue. thank you Andriy Berezhanskyy (@aberezh)Nintorac commentedon Dec 31, 2023
I get a similar issue in a DuckDB udf, my error looks like this
and @cireps workaround works for my situation too
sstoops commentedon Feb 1, 2024
My team ran into this exact issue, though under slightly different circumstances. We need to use the Graph API within the context of an Async Django request. We found the exact same behavior: the first request would succeed and subsequent requests would throw the
RuntimeError: Event loop is closederror.While a few folks here have proposed different workaround to deal with the event loop itself, I wanted figure out why the SDK was hanging on to references to event loops.
Background
Using Async Django as the example, since it is where I ran into this issue, if you're not running a pure async stack (eg. one of your middlewares is non-async), then Django will run each request in a new thread with a new event loop. The Django debug server will also exhibit the same thread-per-request behavior even if you are top-to-bottom async.
This is similar to the behavior described in this issue: calling
asyncio.runmultiple times from a script. I believe this is also the same issue reported in #82.Cause
So something, somewhere in the SDK is grabbing a reference to an event loop, and attempting to use that reference during a subsequent request.
After much sleuthing, I found the pattern leading to this behavior:
GraphServiceClientsource, I found in its__init__method that it constructs aGraphRequestAdapterGraphRequestAdapter, I found the following used as an argument to the__init__:client = GraphClientFactory.create_with_default_middleware(options=options). That line means that it will generate a default client factory at the time this module is imported, not when it is executed.GraphClientFactory.create_with_default_middlewaremethod, we see the same pattern in the arg default:client: httpx.AsyncClient = KiotaClientFactory.get_default_client(). Again, that line is executed when the module is first imported, not when the function is called. That means all instances ofGraphClientFactorywill share the same client object that was created under whichever thread/event loop the module was imported in.To summarize all that, every
GraphServiceClientyou create is going to end up sharing the same underlyinghttpx.AsyncClient. Thathttpx.AsyncClientinstance is constructed when you import the module, not when you actually run any code. This client is what (eventually) holds on to the reference to the event loop.Our workaround
I managed to come up with a workaround that seems to be working so far. It has been tested both in a non-pure-async Django stack (meaning there's a sync middleware causing Django shuttle requests out to threads) and in the Django debug server.
What does this do?
GraphServiceClientcan accept eithercredentialsandscope, resulting in it constructing an auth scope and request adapter for you, or it can just accept an existing request adapter.The
get_request_adapterfunction essentially mimics the existing SDK code, but instead of constructing any clients at the global scope (during import), it constructs a new request adapter each time you construct a client.With this code, you could create a new client within the scope of your async code.
For example:
Note that the two calls to
asyncio.run(...), and therefore two event loops, now succeeds.Addendum
There are many other instances in the SDK where clients are constructed within the scope of a method's default args, meaning, at the global scope. My workaround was the bare minimum I needed to get around the closed event loop issue for my own use case. If you run into this error in other parts of the SDK, I suggest looking for this problematic pattern.
shemogumbe commentedon Apr 23, 2024
Thanks for documenting this,
While this works and solves for your use case, having multiple asyncio runs is considered an antipattern.
Typically, we should have one
asyncio.run(),as it sets up the event loop and starts the asynchronous execution.Besides being considered an antipattern, it can cause unexpected behaviour, note that an event loop manages asynchronous tasks within a single execution context.
To execute within a single loop, use
asyncio.gather()With your workaround, I would consider closing the issue and moving it to a discussion for reference by anyone who might face a similar issue
shemogumbe commentedon Apr 23, 2024
Moved to #683
sstoops commentedon Apr 23, 2024
Shem Ogumbe (@shemogumbe) The multiple calls to
asyncio.run()was to show that the code can be called multiple times without encountering the event loop issue. The issue my team encountered was using a framework where we don't have control over the event loop. Multipleasyncio.run()calls simply emulates this behavior. I will update my comment to clarify.IamMrandrew commentedon May 13, 2024
I used this workaround for my Flask application with async extra, together with creating a new client every time when accepting a request.
It solves the issue where I was getting the
RuntimeError: Event loop is closed erroron my second request after the application starts.Not sure if it affect performance or as anti-pattern. But I couldn't find relevant examples on how should I implement it with this SDK. Let me know if there's update on this.
vschmidt commentedon Aug 28, 2025
Thanks for this, this was the unique way to deal with loop breaks.
I try several ways to deal with my coroutines, and this solve the problem.
Bain-OS commentedon Aug 28, 2025
fix(async): Added workaround for closed event loop
fix(msgraph): Applied workaround for closed event loop
fix(msgraph): Applied workaround for closed event loop (#34)