Skip to content

fix(evaluation): skip built-in tools in the rubric multi-turn trajectory evaluator - #7608

Open
techi101 wants to merge 1 commit into
google:mainfrom
techi101:fix/rubric-trajectory-builtin-tools
Open

techi101 wants to merge 1 commit into
google:mainfrom
techi101:fix/rubric-trajectory-builtin-tools

Conversation

@techi101

@techi101 techi101 commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

Link to Issue or Description of Change

1. Link to an existing issue (if applicable):

N/A, no existing issue.

2. Or, if no issue exists, describe the change:

Problem:
RubricBasedMultiTurnTrajectoryEvaluator crashes when an agent uses a built-in tool such as google_search. Built-in tools are genai Tool objects with function_declarations=None, so the loop that lists the agent's tools raises TypeError: 'NoneType' object is not iterable.

LocalEvalService catches the error and records the metric as NOT_EVALUATED. That status is skipped when the case status is computed, so if another metric passes, the eval case still shows PASSED even though the trajectory metric never ran.

Solution:
Iterate over tool.function_declarations or [], so tools without function declarations are skipped and function tools are listed as before. This is the smallest change that stops the crash and leaves the judge prompt unchanged for function tools.

Testing Plan

Unit Tests:

  • I have added or updated unit tests for my change.
  • All unit tests pass locally.

Added test_app_details_with_built_in_tool: an agent with google_search plus a function tool. It fails on main with the TypeError and passes with the fix.

$ pytest tests/unittests/evaluation -q
994 passed

Manual End-to-End (E2E) Tests:

Not run end-to-end; the failure is in prompt assembly and is covered by the unit test above.

Checklist

  • I have read the CONTRIBUTING.md document.
  • I have performed a self-review of my own code.
  • I have commented my code, particularly in hard-to-understand areas.
  • I have added tests that prove my fix is effective or that my feature works.
  • New and existing unit tests pass locally with my changes.
  • I have manually tested my changes end-to-end.
  • Any dependent changes have been merged and published in downstream modules.

Additional context

I used an AI coding assistant (Claude) to help find and fix this; I reviewed and tested the change myself.

…ory evaluator

Built-in tools such as google_search are genai Tools with no
function_declarations, so listing tools raised TypeError. The eval
service then records the metric as NOT_EVALUATED, which is ignored
when computing the case status, so the case could still pass.

@BichengWang BichengWang left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independently validated d5c1c08cb04c6b23af272346f1ac22f881d85186 on macOS / Python 3.13.16. I checked the discussion and reviews before testing; there was no prior review feedback.

The new regression fails with TypeError: 'NoneType' object is not iterable when only the production module is restored to the PR base ebe11595e7da50a1f158f64e266fb4e73f60bea1 (1 failed, 8 passed). On this head, the owning test file and the rubric evaluator, LLM judge, and judge-utils suites give 120 passed. Changed-file pre-commit checks pass; hooks without applicable files are skipped.

I also exercised the public evaluate_invocations path with two turns and three scripted judge responses per evaluation. Function-only tools, search plus a function, code execution plus a function, and an empty declaration list plus a function all reach scoring with overall_score=1.0, the first turn NOT_EVALUATED, and the last turn PASSED. The judge requests retain both user/assistant turns, the agent instructions, and exactly one copy of the function name and description. Separate no-tool and search-only prompt-assembly controls pass.

This supports the narrow or [] fallback: it prevents a built-in tool from aborting prompt assembly while retaining function declarations. I found no blocking issue in the changed lines. Validation used the checkout's real assembly, prompt, parser, and aggregation code with a scripted judge; no live model, full tox matrix, or complete evaluation suite was run. This does not validate the separate LocalEvalService status-aggregation behavior described in the PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants