Skip to content

fix(models): take the final input and cache counts from Anthropic message_delta - #7624

Open
pcbeingused333 wants to merge 1 commit into
google:mainfrom
pcbeingused333:fix/anthropic-stream-final-usage
Open

pcbeingused333 wants to merge 1 commit into
google:mainfrom
pcbeingused333:fix/anthropic-stream-final-usage

Conversation

@pcbeingused333

Copy link
Copy Markdown

Link to Issue or Description of Change

2. Or, if no issue exists, describe the change:

Problem:

The streaming path of AnthropicLlm reads the input and cache token counts from message_start only and keeps them to the end. message_delta also reports input_tokens, cache_read_input_tokens and cache_creation_input_tokens, cumulative and final, and they were ignored. When message_start could not know the final values, a streamed turn reported fewer prompt tokens than the same turn without streaming.

Solution:

While streaming, keep the uncached input, cache-read and cache-write counts apart, let message_delta replace each count it carries (a field the event leaves out keeps its message_start value), and sum them once into prompt_token_count when the final response is built. When message_delta has no input or cache fields the result is the same as before.

Testing Plan

Unit Tests:

  • I have added or updated unit tests for my change.
  • All unit tests pass locally.

Added test_streaming_takes_final_input_and_cache_counts_from_message_delta: message_start reports 3 input tokens and no cache, message_delta reports 120 input, 30 cache-read and 10 cache-write tokens, and the final response must report a prompt count of 160, 30 cached, 10 cache-creation and a total of 167. pytest tests/unittests/models/test_anthropic_llm.py: 184 passed (anthropic 1.15.0). On main only the new test fails. pyink --check and isort --check are clean on the touched files.

Manual End-to-End (E2E) Tests:

Not run against the real API; the stream events in the test follow the shape of the Anthropic streaming documentation.

Checklist

  • I have read the CONTRIBUTING.md document.
  • I have performed a self-review of my own code.
  • I have commented my code, particularly in hard-to-understand areas.
  • I have added tests that prove my fix is effective or that my feature works.
  • New and existing unit tests pass locally with my changes.
  • I have manually tested my changes end-to-end.
  • Any dependent changes have been merged and published in downstream modules.

Additional context

This change was drafted with Claude Code; the tests above were run before opening the PR.

🤖 Generated with Claude Code

…sage_delta

The streaming path read input and cache token counts from message_start
only. message_delta reports the cumulative input_tokens and cache counts
too, and those are the final ones, so a streamed turn could report fewer
prompt tokens than the non-streaming call. Keep the three counts apart
while streaming, let message_delta override the ones it carries, and sum
them once at the end.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants