Skip to content

CookieJar.add_cookie_header() is linear in the number of stored domains #160027

Open

Description

@Kludex

Feature or enhancement

CookieJar.add_cookie_header() gets slower with every stored domain, even when only one cookie matches the request:

import timeit, urllib.request
from http.cookiejar import CookieJar, Cookie

def cookie(name, domain):
    return Cookie(0, name, "v", None, False, domain, True, False, "/", True, False, None, True, None, None, {})

for n in (10, 100, 1000, 10000):
    jar = CookieJar()
    for i in range(n):
        jar.set_cookie(cookie(f"c{i}", f"site{i}.example"))
    req = urllib.request.Request("https://site0.example/")
    t = min(timeit.repeat(lambda: jar.add_cookie_header(req), number=200, repeat=3)) / 200
    print(f"{n:>6} domains: {t*1e6:8.1f} us")
    10 domains:     37.9 us
   100 domains:    320.8 us
  1000 domains:   3086.9 us
 10000 domains:  30450.6 us

Each call:

  1. Runs DefaultCookiePolicy.domain_return_ok() for every stored domain, and each call re-parses the request URL via eff_request_host().
  2. Runs clear_expired_cookies(), which iterates every stored cookie.

Possible improvements, in increasing size:

  1. Compute eff_request_host() once per request instead of once per domain.
  2. With DefaultCookiePolicy, look up only the domain keys that can match the request host (its dot-suffixes) instead of iterating all domains.
  3. Track the earliest expiry in set_cookie() and skip the sweep until it has passed.

HTTPX2 ships 2 and 3 as a CookieJar subclass, which passes test_http_cookiejar: pydantic/httpx2#1285

I'm happy to send a PR.

Activity

  1. added
    type-featureA feature request or enhancement
    performancePerformance or resource usage
    stdlibStandard Library Python modules in the Lib/ directory
    on Oct 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    performancePerformance or resource usagestdlibStandard Library Python modules in the Lib/ directorytype-featureA feature request or enhancement

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions