Skip to content
This repository was archived by the owner on Aug 30, 2026. It is now read-only.
This repository was archived by the owner on Aug 30, 2026. It is now read-only.

AttributeError: '_UnaryStreamMultiCallable' object has no attribute '_retry' #596

Closed

Description

@louis030195

Environment details

  • OS type and version: MacOS 12.3.1
  • Python version: python --version Python 3.8.13
  • pip version: pip --version pip 21.3.1
  • google-cloud-firestore version: pip show google-cloud-firestore Version: 2.3.4

Steps to reproduce

  1. firebase_admin.initialize_app()
  2. [print(e) for e in firestore.client().collection("foos").stream()]

Stack trace

Traceback (most recent call last):
  File "/Users/louisbeaumont/Documents/inkdropy/env/lib/python3.8/site-packages/google/api_core/grpc_helpers.py", line 97, in __next__
    return next(self._wrapped)
  File "/Users/louisbeaumont/Documents/inkdropy/env/lib/python3.8/site-packages/grpc/_channel.py", line 426, in __next__
    return self._next()
  File "/Users/louisbeaumont/Documents/inkdropy/env/lib/python3.8/site-packages/grpc/_channel.py", line 809, in _next
    raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
        status = StatusCode.INTERNAL
        details = "Received RST_STREAM with error code 2"
        debug_error_string = "{"created":"@1654001431.880353000","description":"Error received from peer ipv6:[2a00:1450:4007:80f::200a]:443","file":"src/core/lib/surface/call.cc","file_line":1070,"grpc_message":"Received RST_STREAM with error code 2","grpc_status":13}"
>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/louisbeaumont/Documents/inkdropy/env/lib/python3.8/site-packages/google/cloud/firestore_v1/query.py", line 279, in stream
    response = next(response_iterator, None)
  File "/Users/louisbeaumont/Documents/inkdropy/env/lib/python3.8/site-packages/google/api_core/grpc_helpers.py", line 100, in __next__
    raise exceptions.from_grpc_error(exc) from exc
google.api_core.exceptions.InternalServerError: 500 Received RST_STREAM with error code 2

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "scripts/hack.py", line 638, in <module>
    fire.Fire(
  File "/Users/louisbeaumont/Documents/inkdropy/env/lib/python3.8/site-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/Users/louisbeaumont/Documents/inkdropy/env/lib/python3.8/site-packages/fire/core.py", line 466, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/Users/louisbeaumont/Documents/inkdropy/env/lib/python3.8/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "scripts/hack.py", line 480, in hack
    for doc in tqdm(fc.collection("products").stream()):
  File "/Users/louisbeaumont/Documents/inkdropy/env/lib/python3.8/site-packages/tqdm/std.py", line 1180, in __iter__
    for obj in iterable:
  File "/Users/louisbeaumont/Documents/inkdropy/env/lib/python3.8/site-packages/google/cloud/firestore_v1/query.py", line 281, in stream
    if self._retry_query_after_exception(exc, retry, transaction):
  File "/Users/louisbeaumont/Documents/inkdropy/env/lib/python3.8/site-packages/google/cloud/firestore_v1/query.py", line 230, in _retry_query_after_exception
    retry = gapic_callable._retry
AttributeError: '_UnaryStreamMultiCallable' object has no attribute '_retry'

Sometimes it happens, sometimes it does not... I know it's not the best issue description, I never had issue with stream() before.

In my for ...stream() I run a bunch of set / delete in batch queries, can it be related (yes I ensure max batch size 500)? Am I hitting the limits of Firestore?

Can you delete documents while using stream()?

Or is stream() kind of in conflict with batch writes?

Read operations must come before write operations
https://firebase.google.com/docs/firestore/manage-data/transactions

Activity

  1. laurahsisson commented on Aug 10, 2022

    @laurahsisson

    Also running into this issue when trying to use stream() on a collection
    My sense is that there is some kind of version conflict between the latest version of google-cloud-firestore and google-api-core. In other words, google-cloud-firestore is calling to some deprecated or remove behavior on the _UnaryStreamMultiCallable.

    Any advice on how to fix or avoid this issue on failed calls?
    This also seems to be popping up on Reddit.

    EDIT:
    I've been able to avoid this issue by just passing in my own google.api_core.retry.Retry object.

    db.collection(collection).stream(retry=retry.Retry(deadline=60))
    

    See the example provided here.

  2. victorkingi commented on Sep 2, 2022

    @victorkingi

    I have this issue as well. I hope a solid fix comes along soon

  3. YannLevavasseur commented on Sep 6, 2022

    @YannLevavasseur

    Previous fix with retry object doesn't work in my case.
    Did someone were able to fix this issue ?

  4. victorkingi commented on Sep 6, 2022

    @victorkingi

    I don't think so, the devs have been slow in replying to this open issue. I found the bug through fuzzing with atheris fuzzer

  5. victorkingi commented on Sep 11, 2022

    @victorkingi

    I found an easy fix before the devs give us an official reply.

    Wrap the call to db.collection("COLLECTION_NAME").stream() in a try except block which excepts AttributeError. In the except block, call db.collection("COLLECTION_NAME").stream() again. Wrap the whole of that in a function and you've got a recursive call to stream everytime it fails. If successful, the function returns the query snapshot.

    You can add a maximum number of retries before reporting the failure.

  6. added
    type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.
    on Dec 1, 2022
  7. efroemling commented on Dec 7, 2022

    @efroemling

    Just started running into this issue myself too. In my case it is from simply streaming through a large collection of documents; no writes/deletes/etc. happening.
    This is on an Ubuntu 18 LTS system with Python 3.10.8, google-api-core 2.11.0, and google-cloud-firestore 2.7.2

  8. rbngz commented on Dec 28, 2022

    @rbngz

    I have the same issue. Some of my production servers suddenly started throwing this exception. Hope this will get fixed soon

  9. added
    priority: p2Moderately-important priority. Fix may not be included in next release.
    on Feb 13, 2023
  10. meredithslota commented on Jun 5, 2023

    @meredithslota
    Contributor

    The current requirements for this library and google-api-core[grpc] are:

        "google-api-core[grpc] >= 1.34.0, <3.0.0dev,!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,!=2.8.*,!=2.9.*,!=2.10.*"
    

    For anyone still experiencing this issue, can you confirm the versions that you're using? https://github.com/googleapis/python-api-core/releases/tag/v2.11.0 contains some fixes that likely help with this issue.

  11. self-assigned this
    on Jun 6, 2023
  12. efroemling commented on Jan 19, 2024

    @efroemling

    Hi. I hadn't seen this error show up in a while so I assumed maybe it was fixed, but I just added some code that is streaming through a large set of documents and it appears I am running into it again. The following is the stack trace from my app-engine app where it is happening, and also the list of pip packages from that same app-engine setup. This includes google-api-core 2.15.0 at this point, so it doesn't seem that 2.11.0 fixed it.

    Is anyone else still seeing this? Should this be reopened?

    Screenshot 2024-01-18 at 5 16 42 PM
    Package                        Version
    ------------------------------ ---------------
    app-store-server-library       1.0.0
    asn1                           2.7.0
    attrs                          23.2.0
    bcrypt                         4.1.2
    blinker                        1.7.0
    boto3                          1.34.20
    botocore                       1.34.21
    CacheControl                   0.13.1
    cachetools                     5.3.2
    cattrs                         23.1.2
    certifi                        2023.11.17
    cffi                           1.16.0
    charset-normalizer             3.3.2
    click                          8.1.7
    cryptography                   41.0.7
    ecdsa                          0.18.0
    enum-compat                    0.0.3
    Flask                          3.0.0
    google-api-core                2.15.0
    google-api-python-client       2.114.0
    google-auth                    2.26.2
    google-auth-httplib2           0.2.0
    google-auth-oauthlib           1.2.0
    google-cloud-appengine-logging 1.4.0
    google-cloud-audit-log         0.2.5
    google-cloud-core              2.4.1
    google-cloud-datastore         2.19.0
    google-cloud-firestore         2.14.0
    google-cloud-logging           3.9.0
    google-cloud-storage           2.14.0
    google-cloud-tasks             2.15.0
    google-cloud-trace             1.12.0
    google-crc32c                  1.5.0
    google-resumable-media         2.7.0
    googleapis-common-protos       1.62.0
    grpc-google-iam-v1             0.13.0
    grpcio                         1.60.0
    grpcio-status                  1.60.0
    gunicorn                       21.2.0
    hiredis                        2.3.2
    httplib2                       0.22.0
    idna                           3.6
    itsdangerous                   2.1.2
    Jinja2                         3.1.3
    jmespath                       1.0.1
    jsonpickle                     3.0.2
    Markdown                       3.5.2
    MarkupSafe                     2.1.3
    msgpack                        1.0.7
    oauthlib                       3.2.2
    packaging                      23.2
    pillow                         10.2.0
    pip                            23.3.1
    proto-plus                     1.23.0
    protobuf                       4.25.2
    pyasn1                         0.5.1
    pyasn1-modules                 0.3.0
    pycparser                      2.21
    pycryptodomex                  3.20.0
    PyJWT                          2.8.0
    pyOpenSSL                      23.3.0
    pyparsing                      3.1.1
    python-dateutil                2.8.2
    PyYAML                         6.0.1
    redis                          5.0.1
    requests                       2.31.0
    requests-oauthlib              1.3.1
    rsa                            4.9
    s3transfer                     0.10.0
    setuptools                     69.0.2
    six                            1.16.0
    types-Markdown                 3.5.0.20240106
    types-protobuf                 4.24.0.20240106
    types-pyOpenSSL                23.3.0.20240106
    types-python-dateutil          2.8.19.20240106
    types-redis                    4.6.0.20240106
    typing_extensions              4.9.0
    ua-parser                      0.18.0
    uritemplate                    4.1.1
    urllib3                        2.0.7
    user-agents                    2.2.0
    Werkzeug                       3.0.1
    wheel                          0.42.0
    
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

🚨This issue needs some love.api: firestoreIssues related to the googleapis/python-firestore API.priority: p2Moderately-important priority. Fix may not be included in next release.triage meI really want to be triaged.type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions