Repository navigation
fix(models): take the final input and cache counts from Anthropic message_delta - #7624
Open
pcbeingused333 wants to merge 1 commit into
Open
pcbeingused333 wants to merge 1 commit into
pcbeingused333 wants to merge 1 commit into
Conversation
…sage_delta The streaming path read input and cache token counts from message_start only. message_delta reports the cumulative input_tokens and cache counts too, and those are the final ones, so a streamed turn could report fewer prompt tokens than the non-streaming call. Keep the three counts apart while streaming, let message_delta override the ones it carries, and sum them once at the end.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Link to Issue or Description of Change
2. Or, if no issue exists, describe the change:
Problem:
The streaming path of
AnthropicLlmreads the input and cache token counts frommessage_startonly and keeps them to the end.message_deltaalso reportsinput_tokens,cache_read_input_tokensandcache_creation_input_tokens, cumulative and final, and they were ignored. Whenmessage_startcould not know the final values, a streamed turn reported fewer prompt tokens than the same turn without streaming.Solution:
While streaming, keep the uncached input, cache-read and cache-write counts apart, let
message_deltareplace each count it carries (a field the event leaves out keeps itsmessage_startvalue), and sum them once intoprompt_token_countwhen the final response is built. Whenmessage_deltahas no input or cache fields the result is the same as before.Testing Plan
Unit Tests:
Added
test_streaming_takes_final_input_and_cache_counts_from_message_delta:message_startreports 3 input tokens and no cache,message_deltareports 120 input, 30 cache-read and 10 cache-write tokens, and the final response must report a prompt count of 160, 30 cached, 10 cache-creation and a total of 167.pytest tests/unittests/models/test_anthropic_llm.py: 184 passed (anthropic 1.15.0). Onmainonly the new test fails.pyink --checkandisort --checkare clean on the touched files.Manual End-to-End (E2E) Tests:
Not run against the real API; the stream events in the test follow the shape of the Anthropic streaming documentation.
Checklist
Additional context
This change was drafted with Claude Code; the tests above were run before opening the PR.
🤖 Generated with Claude Code