Repository navigation
Conversation
…ory evaluator Built-in tools such as google_search are genai Tools with no function_declarations, so listing tools raised TypeError. The eval service then records the metric as NOT_EVALUATED, which is ignored when computing the case status, so the case could still pass.
BichengWang
left a comment
There was a problem hiding this comment.
Independently validated d5c1c08cb04c6b23af272346f1ac22f881d85186 on macOS / Python 3.13.16. I checked the discussion and reviews before testing; there was no prior review feedback.
The new regression fails with TypeError: 'NoneType' object is not iterable when only the production module is restored to the PR base ebe11595e7da50a1f158f64e266fb4e73f60bea1 (1 failed, 8 passed). On this head, the owning test file and the rubric evaluator, LLM judge, and judge-utils suites give 120 passed. Changed-file pre-commit checks pass; hooks without applicable files are skipped.
I also exercised the public evaluate_invocations path with two turns and three scripted judge responses per evaluation. Function-only tools, search plus a function, code execution plus a function, and an empty declaration list plus a function all reach scoring with overall_score=1.0, the first turn NOT_EVALUATED, and the last turn PASSED. The judge requests retain both user/assistant turns, the agent instructions, and exactly one copy of the function name and description. Separate no-tool and search-only prompt-assembly controls pass.
This supports the narrow or [] fallback: it prevents a built-in tool from aborting prompt assembly while retaining function declarations. I found no blocking issue in the changed lines. Validation used the checkout's real assembly, prompt, parser, and aggregation code with a scripted judge; no live model, full tox matrix, or complete evaluation suite was run. This does not validate the separate LocalEvalService status-aggregation behavior described in the PR.
Link to Issue or Description of Change
1. Link to an existing issue (if applicable):
N/A, no existing issue.
2. Or, if no issue exists, describe the change:
Problem:
RubricBasedMultiTurnTrajectoryEvaluatorcrashes when an agent uses a built-in tool such asgoogle_search. Built-in tools aregenaiToolobjects withfunction_declarations=None, so the loop that lists the agent's tools raisesTypeError: 'NoneType' object is not iterable.LocalEvalServicecatches the error and records the metric asNOT_EVALUATED. That status is skipped when the case status is computed, so if another metric passes, the eval case still shows PASSED even though the trajectory metric never ran.Solution:
Iterate over
tool.function_declarations or [], so tools without function declarations are skipped and function tools are listed as before. This is the smallest change that stops the crash and leaves the judge prompt unchanged for function tools.Testing Plan
Unit Tests:
Added
test_app_details_with_built_in_tool: an agent withgoogle_searchplus a function tool. It fails on main with theTypeErrorand passes with the fix.Manual End-to-End (E2E) Tests:
Not run end-to-end; the failure is in prompt assembly and is covered by the unit test above.
Checklist
Additional context
I used an AI coding assistant (Claude) to help find and fix this; I reviewed and tested the change myself.