google-agents-cli-eval

# Advanced Eval Commands

Opt-in commands from the Quality Flywheel. The core loop (`eval run`, and `eval generate` / `eval grade`) lives in SKILL.md.

## `eval analyze`

Runs LLM-based failure clustering and root-cause analysis over a `results_*.json` produced by an eval run. Use when you have 10+ failing cases and want categorized failure modes instead of reading the HTML case-by-case. Supported `--metric` values: `multi_turn_task_success`, `multi_turn_tool_use_quality`.

```bash
# Basic: analyze a results file with default settings
agents-cli eval analyze --eval-result artifacts/grade_results/results_<ts>.json

# Advanced: restrict to a specific metric and cap loss clusters
agents-cli eval analyze \
  --eval-result artifacts/grade_results/results_<ts>.json \
  --metric multi_turn_tool_use_quality \
  --top-k 5 \
  --output artifacts/analysis_<ts>.json
```

## `eval optimize`

Runs ADK GEPA prompt optimization against a target metric. Suitable after an eval run identifies prompt-only failures (wording, not tool/orchestration logic). `--dataset` and `--target-metric` override values in `--config` when both are passed. **Long-running and expensive, see Stage 4 of the Quality Flywheel for usage guidance.**

```bash
# Basic: optimize against a single metric on a dataset
agents-cli eval optimize --dataset tests/eval/datasets/basic-dataset.json --target-metric final_response_quality

# Advanced: drive multi-metric / multi-dataset optimization from a config file
agents-cli eval optimize --config tests/eval/optimization_config.json
```

## `eval submit` / `eval results` (cloud-side)

The managed, asynchronous counterpart to the local path, for large or CI-driven runs: `eval submit` hands the dataset and metrics to the Agent Platform Eval Service, and `eval results` polls and downloads the scores. Pass `--resource-name <agent>` to also run inference server-side (managed `generate` + `grade`); omit it to grade an existing trace (managed `grade`).

```bash
# Grade an existing trace server-side; returns a run resource name to poll
agents-cli eval submit --dataset tests/eval/datasets/basic-dataset.json --dest gs://my-bucket
# Add --resource-name projects/<p>/locations/<l>/reasoningEngines/<id> to run inference too

agents-cli eval results --run-id <run-resource-name>
```

# Evaluating Agents with `google_search` and Built-in Tools

## google_search Behavior (IMPORTANT)

`google_search` is NOT a regular tool — it's a **model-internal grounding feature**.

**Key behavior:**
- Custom tools (`save_preferences`, `save_feedback`) → appear as `function_call` in trajectory
- `google_search` → NEVER appears in trajectory (happens inside the model)

**How google_search works internally:**
```python
llm_request.config.tools.append(
    types.Tool(google_search=types.GoogleSearch())  # Injected into model config
)
```

Search results come back as `grounding_metadata`, not function call/response events. But the evaluator STILL detects it at the session level:
```json
{
  "error_code": "UNEXPECTED_TOOL_CALL",
  "error_message": "Unexpected tool call: google_search"
}
```

This causes `multi_turn_tool_use_quality` to fail for agents whose **only** tool is `google_search` — the evaluator flags an unexpected tool call it can never see in the trace. For agents that also call function tools, the metric still scores those function-tool calls (see metric compatibility below).

**Metric compatibility for `google_search` agents:**

| Metric | Usable? | Why |
|--------|---------|-----|
| `multi_turn_tool_use_quality` | NO | Always fails due to unexpected google_search (the `google_search` invocation is detected by the evaluator but never appears as a `function_call` / `function_response` event) |
| `final_response_quality` | YES | Adaptive rubric-based evaluation; works without a reference answer |
| `final_response_match` | NO | Search results vary across runs, so the agent's response rarely matches a fixed reference |

**Dataset best practices for `google_search` agents:**

```json
{
  "eval_cases": [
    {
      "eval_case_id": "news_digest_test",
      "prompt": {
        "role": "user",
        "parts": [{"text": "Give me my news digest."}]
      }
      // NO trajectory criteria for google_search - it won't appear in the trace anyway
    }
  ]
}
```

For agents that mix `google_search` with custom function tools, grade the custom tool usage with `multi_turn_tool_use_quality` — it judges the tool calls in the generated trace, so you don't hand-author expected calls. Optionally add a `reference` response for reference-based matching:
```json
{
  "eval_case_id": "news_digest_feedback",
  "prompt": {
    "role": "user",
    "parts": [{"text": "Great, save my positive feedback."}]
  },
  "reference": {
    "response": {
      "role": "model",
      "parts": [{"text": "Feedback saved!"}]
    }
  }
}
```
The `google_search` invocation still won't appear in the trace, so `multi_turn_tool_use_quality` only assesses the function-tool calls (e.g., `save_feedback`).

**Config for `google_search` agents (`eval_config.yaml`):**

```yaml
metrics_to_run:
  - final_response_quality
```

The built-in `final_response_quality` is sufficient for most `google_search` agents; it auto-generates a content-based rubric. Define a custom override in `custom_metrics` only if you need project-specific judge instructions — see SKILL.md's *Evaluation Configuration Schema* for the override pattern.

**Bottom line:** `google_search` is a model feature, not a function tool. You cannot test it with trajectory matching. Use `final_response_quality` to verify the agent produces grounded, cited responses.

---

## ADK Built-in Tools: Trajectory Behavior Reference

**Model-Internal Tools (DON'T appear in trajectory):**

| Tool | In Trajectory? | Eval Strategy |
|------|----------------|---------------|
| `google_search` | No | Rubric-based |
| `google_search_retrieval` | No | Rubric-based |
| `BuiltInCodeExecutor` | No | Check output |
| `VertexAiSearchTool` | No | Rubric-based |
| `url_context` | No | Rubric-based |

These inject into `llm_request.config.tools` as model capabilities:
```python
types.Tool(google_search=types.GoogleSearch())
types.Tool(code_execution=types.ToolCodeExecution())
types.Tool(retrieval=types.Retrieval(...))
```

**Function-Based Tools (DO appear in trajectory):**

| Tool | In Trajectory? | Eval Strategy |
|------|----------------|---------------|
| `load_web_page` | Yes | `multi_turn_tool_use_quality` works |
| Custom tools | Yes | `multi_turn_tool_use_quality` works |
| AgentTool | Yes | `multi_turn_tool_use_quality` works |

These generate `function_call` and `function_response` events:
```python
types.Tool(function_declarations=[...])
```

**Quick Reference — Can I use `multi_turn_tool_use_quality`?**
- `google_search` → NO (model-internal)
- `code_executor` → NO (model-internal)
- `VertexAiSearchTool` → NO (model-internal)
- `url_context` → NO (model-internal)
- `load_web_page` → YES (FunctionTool)
- Custom functions → YES (FunctionTool)

**When mixing both types** (e.g., `google_search` + `save_preferences`):
1. Rely on `final_response_quality` for overall quality, OR
2. Keep `multi_turn_tool_use_quality` — it assesses the function-tool calls that do appear in the trace, accepting that the `google_search` step is invisible to it

**Rule of Thumb:**
- If a tool provides grounding/retrieval/execution capabilities built into Gemini → model-internal, won't appear in trajectory
- If it's a Python function you can call → appears in trajectory

### Model thinking mode may bypass tools

Models with "thinking" enabled may decide they have sufficient information and skip tool calls. Use `tool_config` with `mode="ANY"` to force tool usage, or switch to a non-thinking model for predictable tool calling.

### Mock mode for external APIs

When your agent calls external APIs, add mock mode so evals can run without real credentials:
```python
def call_external_api(query: str) -> dict:
    api_key = os.environ.get("EXTERNAL_API_KEY", "")
    if not api_key or api_key == "dummy_key":
        return {"status": "success", "data": "mock_response"}
    # Real API call here
```

# Evaluation Dataset Schema

Canonical formats for evaluation datasets in the Agent Platform Evaluation
SDK. The summary below covers the type tree as of the version this skill
targets — for the live, authoritative definitions see the public SDK source:
[`types/evals.py`](https://github.com/googleapis/python-aiplatform/blob/main/agentplatform/_genai/types/evals.py)
and [`types/common.py`](https://github.com/googleapis/python-aiplatform/blob/main/agentplatform/_genai/types/common.py).

## Core Types

```
EvaluationDataset
└── eval_cases: list[EvalCase]       # List of evaluation cases

EvalCase
├── prompt: Content                          # Single-turn: the user query
├── responses: list[ResponseCandidate]       # Single-turn: model response(s); list to support multi-candidate eval
├── reference: ResponseCandidate             # Ground truth, needed by `final_response_match`
├── context: str | Content                   # Source text, needed by `grounding`
├── agent_data: AgentData                    # Multi-turn: full conversation trajectory
├── rubric_groups: dict[str, RubricGroup]    # Per-case rubrics; graded by managed rubric metrics
└── (extra fields allowed)                   # Custom fields for custom metrics

ResponseCandidate
└── response: Content                # The actual Content (role + parts)

AgentData
├── agents: dict[str, AgentConfig]   # Agent definitions
└── turns: list[ConversationTurn]    # Ordered conversation turns

ConversationTurn
├── turn_index: int                  # 0-based turn number
└── events: list[AgentEvent]         # Events within this turn

AgentEvent
├── author: str                      # "user", agent_id, or "tool"
└── content: Content                 # Content with role and parts
```

> **Note on `responses` and `reference`.** Both wrap a `Content` inside a `ResponseCandidate` object. So a single-turn case writes `"responses": [{"response": {"role": "model", "parts": [...]}}]` and `"reference": {"response": {"role": "model", "parts": [...]}}` — NOT a bare `Content`. `prompt` and `agent_data.turns[].events[].content` are bare `Content` (not wrapped).

## Single-Turn Dataset

For simple prompt-response evaluation (e.g., QA, summarization).

```json
{
  "eval_cases": [
    {
      "eval_case_id": "capital_of_france",
      "prompt": {
        "role": "user",
        "parts": [{"text": "What is the capital of France?"}]
      },
      "responses": [
        {
          "response": {
            "role": "model",
            "parts": [{"text": "The capital of France is Paris."}]
          }
        }
      ],
      "reference": {
        "response": {
          "role": "model",
          "parts": [{"text": "Paris"}]
        }
      }
    },
    {
      "eval_case_id": "summarize_article",
      "prompt": {
        "role": "user",
        "parts": [{"text": "Summarize this article: ..."}]
      },
      "responses": [
        {
          "response": {
            "role": "model",
            "parts": [{"text": "The article discusses..."}]
          }
        }
      ]
    }
  ]
}
```

### Required fields by metric type

| Metric category | Required fields |
|---|---|
| Predefined (single-turn) | `prompt`, `responses` |
| Computation-based | `responses`, `reference` |
| Translation | `prompt` (source), `responses`, `reference` |
| Custom LLM/code | Fields referenced in your template/function |

## Multi-Turn / Multi-Agent Dataset

For evaluating multi-turn agent conversations, including systems with
multiple collaborating agents and tool calls. The `agents` map declares
all participating agents; `turns` is the chronological conversation,
where each `event` author is `"user"`, an agent ID from the `agents`
map, or `"tool"`.

```json
{
  "eval_cases": [
    {
      "eval_case_id": "flight_booking_via_specialist",
      "agent_data": {
        "agents": {
          "router": {
            "agent_id": "router",
            "agent_type": "RouterAgent",
            "instruction": "Route requests to the appropriate specialist."
          },
          "flight_bot": {
            "agent_id": "flight_bot",
            "agent_type": "SpecialistAgent",
            "instruction": "Search and book flights.",
            "tools": [{
              "function_declarations": [{
                "name": "search_flights",
                "description": "Search flights by destination",
                "parameters": {
                  "type": "OBJECT",
                  "properties": {
                    "destination": {"type": "STRING"}
                  }
                }
              }]
            }]
          }
        },
        "turns": [
          {
            "turn_index": 0,
            "events": [
              {
                "author": "user",
                "content": {
                  "parts": [{"text": "Book a flight to NYC"}]
                }
              },
              {
                "author": "router",
                "content": {
                  "parts": [{"text": "Routing to flight_bot."}]
                }
              }
            ]
          },
          {
            "turn_index": 1,
            "events": [
              {
                "author": "flight_bot",
                "content": {
                  "parts": [{
                    "function_call": {
                      "name": "search_flights",
                      "args": {"destination": "NYC"}
                    }
                  }]
                }
              },
              {
                "author": "flight_bot",
                "content": {
                  "parts": [{
                    "function_response": {
                      "name": "search_flights",
                      "response": {"flights": [{"id": "AA123", "price": 320}]}
                    }
                  }]
                }
              },
              {
                "author": "flight_bot",
                "content": {
                  "parts": [{"text": "Found AA123 to NYC for $320."}]
                }
              }
            ]
          }
        ]
      }
    }
  ]
}
```

For a single-agent multi-turn case, omit the extra agent definitions
and use one entry in `agents`.

## Per-Case Rubrics (`rubric_groups`)

`EvalCase.rubric_groups` attaches case-specific criteria, graded one pass/fail verdict per rubric by a managed rubric metric (see `metrics-guide.md`). Write them on the inference-input dataset; `eval generate` carries them onto the trace.

```json
{
  "eval_cases": [
    {
      "eval_case_id": "booking_confirmation",
      "prompt": {"role": "user", "parts": [{"text": "Book my flight to Paris."}]},
      "rubric_groups": {
        "booking_rubrics": {
          "rubrics": [
            {"rubric_id": "confirmation_check", "content": {"property": {"description": "The model must confirm the booking and provide a reference number."}}}
          ]
        }
      }
    }
  ]
}
```

List a managed rubric metric in `metrics_to_run`; with more than one group per case, select it with `metric_spec_parameters.rubric_group_key` (see *Managed Metric Parameters* in `metrics-guide.md`). Results carry `rubric_verdicts` per metric (`evaluated_rubric.rubric_id`, `verdict`, `reasoning`); the score is the fraction passed.

Service constraints:

- 400 when the key is not on the case: `rubric_group_key '<name>' not found in instance.rubric_groups`.
- 400 with more than one group and no key: `Multiple rubric groups provided in instance but no rubric_group_key specified in metric spec`.
- Single-turn only: a single-turn metric on a multi-turn trace 400s with `Single-turn metric '<name>_v1' received agent_eval_data with N turns`, and `multi_turn_task_success` accepts the key but grades its own rubrics (hash IDs). Grade multi-turn criteria with a local `custom_function_file` judge (`metrics-guide.md`); it receives `rubric_groups` in `instance`.
- Metrics apply to every case, so split single-turn and multi-turn cases into separate dataset + config pairs.

## Common Mistakes

| Mistake | Fix |
|---|---|
| Using `role="assistant"` | Use `role="model"` (Vertex convention) |
| Missing `turn_index` | Always set sequential 0-based indices |
| Tool response without `function_response` | Wrap in a `function_response` part |
| Using `prompt` field for multi-turn | Use `agent_data` with the full trajectory |
| Mixing `prompt` and `agent_data` in one case | Use one or the other per `EvalCase` |

# Evaluation Metrics Reference

> File paths below reference the scaffolded layout (`tests/eval/eval_config.yaml` or `.json`). Adjust for your project structure if not using `google-agents-cli-scaffold`.

## Managed (Built-in) Metrics Reference

Run `agents-cli eval metric list` for the live set. **Single-turn only** below means the metric 400s on a trace with 2+ turns (`Single-turn metric '<name>_v1' received agent_eval_data with N turns`). The single-turn adaptive-rubric metrics grade a case's own `rubric_groups` instead of generating their own when it supplies them (see *Managed Metric Parameters*).

### Agent metrics (adaptive rubrics)

| Metric ID | Evaluates | Trace |
|-----------|-----------|-------|
| `multi_turn_task_success` | User goal/intent fulfillment across the conversation. Ignores supplied `rubric_groups`. | any |
| `multi_turn_trajectory_quality` | Step sequencing, efficiency, error recovery. | any |
| `multi_turn_tool_use_quality` | Technical and semantic correctness of tool calls. | any |
| `final_response_quality` | Final response plus intermediate tool usage. | single-turn only |
| `final_response_reference_free` | Final response quality with no reference answer. Needs `rubric_groups` on the case (500s without). | single-turn only |
| `tool_use_quality` | Tool selection, parameter accuracy, step order. Needs `function_call` events in the trace. | single-turn only |

> `multi_turn_general_quality` and `multi_turn_text_quality` need a `conversation_history` field that `eval generate` does not produce, and 400 on agent traces. Use `multi_turn_task_success` or `multi_turn_trajectory_quality`.

### General quality metrics (adaptive rubrics, single-turn only)

| Metric ID | Evaluates |
|-----------|-----------|
| `general_quality` | Overall quality with auto-generated criteria. Best starting point for non-agent eval. |
| `text_quality` | Fluency, coherence, grammar. |
| `instruction_following` | Adherence to the constraints in the prompt. |

### Static rubric metrics (fixed criteria, single-turn only)

| Metric ID | Evaluates |
|-----------|-----------|
| `hallucination` | Segments the response into atomic claims and checks each against tool output. |
| `final_response_match` | Judge-scored semantic match against a golden answer, not string equality. Needs `reference` on the case. |
| `grounding` | Labels each sentence of the response supported or contradictory against context. Needs `context` (a string or `Content`) on the case. |
| `safety` | Policy compliance (PII, hate speech, dangerous content, harassment, sexual). |

---

## Custom Metrics

Custom metrics are declared in `eval_config.yaml` (or `.json`) under `custom_metrics`. See SKILL.md's *Evaluation Configuration Schema* section for how `metrics_to_run` selects from the pool. The schema below defines the per-entry fields.

Code-based metrics default to **local in-process execution** (no GCP project or region required); opt into the Vertex AI sandbox with `execution: "remote"`.

> **Scaffolded default metric.** The scaffolded `eval_config.yaml` ships `custom_response_quality` as a local LLM-judge in `tests/eval/response_quality.py` (referenced via `custom_function_file`, run in-process via `google-genai`). It grades on either backend — `genai.Client()` uses `GEMINI_API_KEY` (AI Studio) or ADC (Vertex) — and reads each case's `reference` (ground truth) when present. To grade with the managed Vertex eval service instead, replace it with a built-in metric or an `LLMMetric` (`prompt_template`).

### Example

```yaml
metrics_to_run:
  - multi_turn_trajectory_quality
  - project_response_rubric
  - agent_turn_count

custom_metrics:
  - name: project_response_rubric
    prompt_template: |
      Rate the agent's response 1-5 for helpfulness and accuracy.
      Prompt: {prompt}
      Final response: {response}
      Full trace (for tool-call and reasoning context): {agent_data}
      Return JSON: {"score": <1|2|3|4|5>, "explanation": "<reason>"}
    judge_model_sampling_count: 3

  - name: agent_turn_count
    custom_function: |
      def evaluate(instance):
          turns = (instance.get("agent_data") or {}).get("turns", [])
          return {'score': len(turns)}

  - name: tool_call_count
    execution: remote
    custom_function: |
      def evaluate(instance):
          n = 0
          for turn in (instance.get("agent_data") or {}).get("turns", []):
              for event in turn.get("events", []):
                  for part in (event.get("content") or {}).get("parts", []):
                      if "function_call" in part:
                          n += 1
          return {'score': n}
```

Metrics receive the eval case's `{prompt}`, `{response}`, and `{agent_data}` (and `{reference}` / `{context}` when the case populates them) — see SKILL.md's *Evaluation Configuration Schema → Agent trace field model* for details.

### Schema reference

Each entry in `custom_metrics` must conform to one of two Agent Platform evaluation metric schemas. `custom_function` or `custom_function_file` selects the code-based schema (in-process by default, `CodeExecutionMetric` with `execution: remote`); otherwise it's `LLMMetric`. An entry that carries neither, and whose `name` is a built-in metric, is a *managed metric parameterization* instead (see below).

#### Code Execution Metric (`CodeExecutionMetric`)

Evaluates responses using custom Python code.

| Field | Required | Description |
|-------|----------|-------------|
| `name` | yes | Unique identifier for the metric. |
| `custom_function` | one of | Python source containing `def evaluate(instance):`. Receives an evaluation instance, returns a numeric score or a `{'score', 'explanation'}` dict. |
| `custom_function_file` | one of | Path to a `.py` file containing `def evaluate(instance):`, **resolved relative to the eval config file's directory** (absolute paths honored). Keeps the metric a real, lintable/testable module instead of an inline blob. Mutually exclusive with `custom_function`. Works with both `execution` modes (for `remote`, the file's source is uploaded). |
| `execution` | no | Where the function runs. `"local"` (default) — executed in the CLI process; no GCP project or region required; **runs with the CLI's privileges**, so only use trusted code. `"remote"` — uploaded and executed inside Vertex AI's `CodeExecutionMetric` sandbox; requires a configured GCP project + region. |

**Minimal `custom_function_file` example** — point the metric at a sibling `.py` file instead of an inline blob:

```yaml
# tests/eval/eval_config.yaml
metrics_to_run:
  - turn_count
custom_metrics:
  - name: turn_count
    custom_function_file: metrics.py   # resolved next to this config file
```

```python
# tests/eval/metrics.py  (same directory as the config)
def evaluate(instance):
    turns = (instance.get("agent_data") or {}).get("turns", [])
    return {"score": len(turns)}
```

Run with `agents-cli eval run --config tests/eval/eval_config.yaml`.

**LLM judge in a custom function**: the way to combine your own judge prompt with per-case criteria, and how to grade multi-turn `rubric_groups`:

```python
# tests/eval/rubric_judge.py: keep execution local: the remote sandbox has no network
import json

from google import genai


def evaluate(instance):
    rubrics = [
        r["content"]["property"]["description"]
        for g in (instance.get("rubric_groups") or {}).values()
        for r in g["rubrics"]
    ]
    prompt = (
        f"Criteria: {rubrics}\nConversation: {json.dumps(instance['agent_data'])}\n"
        'Return JSON: {"score": <fraction of criteria met>, "explanation": "<what failed>"}'
    )
    out = genai.Client().models.generate_content(
        model="gemini-3.7-flash",
        contents=prompt,
        config={"response_mime_type": "application/json"},
    )
    return json.loads(out.text)
```

#### LLM-as-a-Judge Metric (`LLMMetric`)

Evaluates responses using an LLM judge driven by a prompt template.

| Field | Required | Description |
|-------|----------|-------------|
| `name` | yes | Unique identifier for the metric. |
| `prompt_template` | yes | Prompt template used by the judge model. With agents-cli's file-based `EvaluationDataset` use `{prompt}`, `{response}`, and `{agent_data}` (the full trajectory). `{reference}` and `{context}` resolve only when the eval case has those fields populated. |
| `rubric_group_name` | n/a | **Rejected by agents-cli.** It makes the service demand rubric verdicts a custom prompt cannot emit (`400 No rubric verdicts found in LLM response`). Grade `rubric_groups` with a managed metric plus `metric_spec_parameters.rubric_group_key`. |
| `judge_model` | no | Judge model (e.g., `gemini-3.7-flash`). |
| `judge_model_sampling_count` | no | Number of judge samples to compute the score (1–32). |
| `judge_model_system_instruction` | no | System instruction for the judge model. |
| `judge_model_generation_config` | no | Generation config for the judge LLM (e.g., `temperature`). |

#### Managed Metric Parameters (`metric_spec_parameters`)

Parameters for a built-in metric, passed through a `custom_metrics` entry that has no `prompt_template` and no `custom_function`. Use a metric ID from the tables above. `rubric_group_key` picks which of the case's `rubric_groups` to grade against, and is required only when a case defines more than one.

```yaml
metrics_to_run:
  - final_response_quality
custom_metrics:
  - name: final_response_quality
    metric_spec_parameters:
      rubric_group_key: case_criteria
```

# Multimodal Evaluation

Two distinct cases are covered here:

1. **Evaluate generated image / video quality** against a text prompt.
2. **Evaluate an agent that consumes multimodal input and produces text** (e.g., the agent describes an image and we want to verify the description).

Both cases use a custom `LLMMetric` with a vision-capable judge model. The built-in adaptive metrics only inspect `text` parts, so they can't reason about media content directly — a custom metric is required for true multimodal grading.

> **Multimodal field-model note.** `agents-cli eval generate` populates `{response}` by extracting the **text** parts of the agent's final event. If your agent returns non-text parts (e.g., `inline_data` images, `file_data` URIs), those parts are not copied into `{response}` automatically. To grade with the full multimodal Content, either hand-author the eval case with a `responses[0].response` Content containing the media parts, or post-process the generated trace file to copy the media parts into `responses`.

> File paths below reference the scaffolded layout (`tests/eval/`). Adjust for your project structure if not using `google-agents-cli-scaffold`.

---

## Dataset shape for multimodal parts

Multimodal content lives inside `parts` as either `inline_data` (base64-encoded bytes with a mime type) or `file_data` (GCS URI reference). Use whichever fits — `file_data` is preferred for anything larger than a few KB.

```json
{ "inline_data": { "mime_type": "image/png", "data": "<base64>" } }
```

```json
{ "file_data": { "mime_type": "image/jpeg", "file_uri": "gs://my-bucket/photos/test.jpg" } }
```

---

## Case 1: Evaluate generated image / video against a text prompt

The eval case has the user prompt as text and the model response as a Content with a media `file_data` (or `inline_data`) part.

```json
{
  "eval_cases": [
    {
      "eval_case_id": "coffee_image",
      "prompt": {
        "role": "user",
        "parts": [{"text": "steaming cup of coffee and a croissant on a table"}]
      },
      "responses": [
        {
          "response": {
            "role": "model",
            "parts": [
              {"file_data": {"mime_type": "image/png", "file_uri": "gs://cloud-samples-data/generative-ai/evaluation/images/coffee.png"}}
            ]
          }
        }
      ]
    }
  ]
}
```

For video, swap `mime_type` to `video/mp4` (or appropriate) and point at a video URI.

### Custom metric (`eval_config.yaml`)

```yaml
custom_metrics:
  - name: image_prompt_alignment
    prompt_template: |
      You are evaluating whether the generated image (in {response}) matches
      the user's text prompt. Consider object presence, attributes, actions,
      composition, and style.

      Prompt: {prompt}
      Image: {response}

      Return JSON: {"score": <0.0-1.0>, "explanation": "<reason>"}
    judge_model: gemini-3.7-flash
    judge_model_sampling_count: 3
```

Run with `agents-cli eval grade --config tests/eval/eval_config.yaml`. For video evaluation, use the same pattern with a video-capable judge model and rubric criteria (motion consistency, temporal coherence, scene transitions).

---

## Case 2: Agent consumes multimodal input, produces text

The user input contains an image / audio / file; the agent produces a text response. To verify the text against the original media (e.g., "did the agent correctly describe this image?"), use a custom `LLMMetric` with a vision-capable judge.

### Dataset shape

The multimodal input lives in the `prompt` field for single-turn, or inside the user-authored event in `agent_data` for multi-turn:

```json
{
  "eval_cases": [
    {
      "eval_case_id": "describe_chart",
      "prompt": {
        "role": "user",
        "parts": [
          {"text": "Describe this image"},
          {"inline_data": {"mime_type": "image/png", "data": "<base64>"}}
        ]
      },
      "responses": [
        {
          "response": {
            "role": "model",
            "parts": [{"text": "The image shows a bar chart..."}]
          }
        }
      ]
    }
  ]
}
```

### Custom metric (`eval_config.yaml`)

```yaml
custom_metrics:
  - name: multimodal_response_quality
    prompt_template: |
      You are evaluating whether the agent's text response accurately reflects
      the user's multimodal input. Inspect the user input parts (which may
      include images, audio, or files) and the agent response, then return JSON:
      {"score": <0.0-1.0>, "explanation": "<reason>"}.

      User input: {prompt}
      Agent response: {response}
    judge_model: gemini-3.7-flash
    judge_model_sampling_count: 3
```

Run with `agents-cli eval run --config tests/eval/eval_config.yaml`.

---

## Notes

- **Built-in adaptive metrics (`final_response_quality`, etc.) skip media parts.** They extract only `.text` parts when constructing the judge prompt. Use a custom `LLMMetric` for true multimodal grading.
- **Choose a vision-capable `judge_model`.** `gemini-3.7-flash` handles images and video; verify capability before relying on it.
- **Sampling count** (`judge_model_sampling_count`) of 3–5 reduces variance for multimodal judges, which can be noisier than text-only.

For the full custom-metric field reference, see `references/metrics-guide.md`. For dataset schema and the `inline_data` / `file_data` part types, see `references/dataset_schema.md`.

# User Simulation for Dynamic Evaluation

> File paths below reference the scaffolded layout. Adjust for your project structure if not using `/google-agents-cli-scaffold`.

## When to Use

Use user simulation when fixed prompts are impractical — the agent may ask for information in different orders or respond in unexpected ways. Instead of hand-recording every user/agent turn, let `agents-cli eval dataset synthesize` ask the Vertex AI evaluation service to generate **user scenarios** for your agent and then play each scenario against an LLM-backed user simulator. The resulting traces (with full `agent_data.turns` populated) drop straight into `agents-cli eval grade`.

A user scenario is a `starting_prompt` (the user's opening message) plus a free-text `conversation_plan` (how the simulated user should behave for the rest of the conversation). You don't author these yourself in the agents-cli flow — `eval dataset synthesize` generates them from your agent's tools and instructions.

> **For deterministic, hand-authored eval cases** (e.g., regression coverage), use the recorded-turns format instead: write `agent_data.turns` directly in your dataset and run `agents-cli eval generate` to play it back. See `references/dataset_schema.md`. `agents-cli eval generate` requires either a top-level `prompt` or `agent_data` on every case; it does **not** play hand-authored `user_scenario` cases.

---

## Running `eval dataset synthesize`

```bash
# Synthesize 3 scenarios (default), simulate them, write traces to artifacts/traces/traces_<ts>.json
agents-cli eval dataset synthesize

# Steer scenario generation with an instruction and environment context
agents-cli eval dataset synthesize \
  -n 5 \
  --max-turns 8 \
  --instruction "Customer asking about refunds" \
  --environment-context "E-commerce support; orders are visible by order_id"

# Use a custom model for scenario generation (default: service default)
agents-cli eval dataset synthesize --model gemini-2.5-pro
```

CLI flags exposed by `agents-cli eval dataset synthesize`:

| Flag | What it controls |
|------|------------------|
| `-n / --count` | Number of scenarios to generate (default 3) |
| `--instruction` | Natural-language steering for scenario generation |
| `--environment-context` | World context the simulator can rely on (e.g., available data) |
| `--model` | Model used for **scenario generation** (server-side; not the simulated user model) |
| `--max-turns` | Cap on user↔agent turns per scenario (default 5) |
| `-o / --output` | Output path; defaults to `artifacts/traces/traces_<ts>.json` |

`synthesize` runs your agent locally and reads its config from the agent's `.env` (the whole file — `GOOGLE_GENAI_USE_VERTEXAI`, `GEMINI_API_KEY`, `GOOGLE_CLOUD_*`, app vars); there are no `--project` / `--region` flags. On Vertex AI, `GOOGLE_CLOUD_LOCATION` also picks the endpoint for the **server-side scenario-generation** call, which only supports a subset of eval regions — keep it `global` (the scaffold default) unless you know your region is supported.

**Simulator internals are NOT user-configurable from agents-cli.** The LLM-backed user simulator that plays the user side runs inside `_synthesize_runner.py` with hardcoded ADK defaults (`gemini-2.5-flash` for the user voice, default thinking config, no `custom_instructions`). Only `--max-turns` reaches it (as `LlmBackedUserSimulatorConfig.max_allowed_invocations`). There is no `eval_config.yaml` key, no `--simulator-model` flag, and no way to override `custom_instructions` or `model_configuration` short of editing `_synthesize_runner.py` directly.

---

## What `synthesize` writes

A single JSON `EvaluationDataset` file at the output path. Each case has:

- `eval_case_id` — server-generated UUID
- `user_scenario` — the generated `{starting_prompt, conversation_plan}` (preserved for traceability)
- `agent_data.turns` — the full simulated conversation: user events, agent responses, tool calls, tool responses

Because `agent_data.turns` is fully populated, the file is already a graded-ready trace. Skip `eval generate` and go straight to `eval grade`:

```bash
agents-cli eval dataset synthesize
agents-cli eval grade   # reads artifacts/traces/ by default
```

If `synthesize` fails for some scenarios, the failing cases land in the output with empty `agent_data.turns` and a stderr warning; the rest still pass through to `eval grade`.

---

## Compatible Metrics

Synthesized traces are multi-turn and have no ground-truth response, so only the three multi-turn metrics apply (every other built-in 400s on a multi-turn trace, and reference-based metrics have nothing to match against):

| Metric | Why it works |
|--------|--------------|
| `multi_turn_task_success` | Adaptive rubric judges whether the simulated user's goal was met |
| `multi_turn_trajectory_quality` | Adaptive rubric on agent reasoning across turns |
| `multi_turn_tool_use_quality` | Adaptive rubric on tool calls across turns |

Example `tests/eval/eval_config.yaml` for grading synthesized traces:

```yaml
metrics_to_run:
  - multi_turn_task_success
  - multi_turn_trajectory_quality
  - multi_turn_tool_use_quality
```

Run with:

```bash
agents-cli eval grade --config tests/eval/eval_config.yaml
```

The `eval_config.yaml` file is read by `eval run`, `eval grade`, and `eval submit`. `eval dataset synthesize` ignores it.

---

## Notes

- **Scenario quality depends entirely on agent metadata.** `generate_conversation_scenarios` reads your agent's instructions and tool descriptions to generate plausible user behaviors. Vague tool descriptions produce vague scenarios. Tighten tool docstrings before running synthesize on a new agent.
- **`--max-turns` is a hard cap.** The simulated user can stop earlier (when its goal is met or it gives up); `--max-turns` only prevents runaway loops.
- **Re-running synthesize generates new scenarios.** There is no seed flag — each invocation produces fresh scenarios. For repeatable regression coverage, write `agent_data.turns` directly (see `references/dataset_schema.md`) instead of relying on `synthesize`.

---
name: google-agents-cli-eval
description: >
  This skill should be used when the user wants to "run an evaluation",
  "evaluate my ADK agent", "write an eval dataset", "analyze eval failures",
  "compare eval results", "optimize agent", or needs guidance on the Agent Platform
  eval methodology and the Quality Flywheel.
  Covers eval metrics, dataset schema, LLM-as-judge scoring, and common failure causes.
  Do NOT use for API code patterns (use google-agents-cli-adk-code), deployment
  (use google-agents-cli-deploy), or project scaffolding (use google-agents-cli-scaffold).
metadata:
  author: Google
  license: Apache-2.0
  version: 1.4.2
  requires:
    bins:
      - agents-cli
    install: "uv tool install google-agents-cli"
---

# Agent Evaluation Guide

> **Requires:** `agents-cli` (`uv tool install google-agents-cli`) — [install uv](https://docs.astral.sh/uv/getting-started/installation/index.md) first if needed.

> **Scaffolded project?** If you used `/google-agents-cli-scaffold`, you already have `agents-cli eval run` (chains `generate` + `grade`), `tests/eval/datasets/`, and `tests/eval/eval_config.yaml`. Start with executing `eval run` and iterate from there.

## Reference Files

| File | Contents |
|------|----------|
| `references/dataset_schema.md` | Canonical EvaluationDataset schema — all field types, JSON examples for single-turn / multi-turn / multi-agent, common mistakes |
| `references/metrics-guide.md` | Complete metrics reference — all built-in metrics, match types, custom metrics, judge model config |
| `references/user-simulation.md` | Dynamic conversation testing — `eval dataset synthesize` flags, what scenarios are, compatible metrics |
| `references/builtin-tools-eval.md` | google_search and model-internal tools — trajectory behavior, metric compatibility |
| `references/advanced-commands.md` | Opt-in commands: `eval analyze`, `eval optimize`, `eval submit` / `eval results` |
| `references/multimodal-eval.md` | Multimodal inputs — eval dataset schema, built-in metric limitations, custom evaluator pattern |

---

## The Quality Flywheel

Improving agent quality is iterative. The 4 stages below describe the loop. Each stage has a Default path (you, the coding agent, do the work directly) and an Opt-in CLI command that delegates to the Agent Platform Eval Service for better quality and scale.

### 1. Prepare Data

**Default:** Use or edit the scaffolded `tests/eval/datasets/basic-dataset.json` to define single-turn eval inputs. Start with 1–2 cases.

**Opt-in:** `agents-cli eval dataset synthesize`: user-simulate multi-turn datasets when you lack data; its output already includes traces, so Stage 2 collapses to `agents-cli eval grade` alone. See *Eval Commands* and `references/user-simulation.md`.

### 2. Run the Eval (always run)

**Default:** `agents-cli eval run` runs the agent over the dataset and grades the traces, writing `results_<ts>.{json,html}` to `artifacts/grade_results/`.

**Decoupled form:** `eval generate` then `eval grade`, for a custom traces location, re-grading without re-running the agent, or traces from `synthesize` (`eval grade` alone).

### 3. Analyze Failures

**Default:** Open the latest `artifacts/grade_results/results_<ts>.html` (or `.json`) and identify failed metrics — see *What to fix when scores fail* below for the fix table.

**Opt-in:** `agents-cli eval analyze`, LLM-based failure clustering; prefer when you have 10+ failing cases and want categorized failure modes. See `references/advanced-commands.md`.

### 4. Optimize & Code Fix

**Default:** Edit the agent — adjust prompts, tool descriptions, instructions, or eval dataset based on the failure analysis. See *What to fix when scores fail* below for the failure → fix mapping.

**Opt-in:** `agents-cli eval optimize` runs ADK GEPA prompt optimization against a target metric (see `references/advanced-commands.md`). Suitable for prompt-only failures. The optimized prompt appears in the command output; capture it and apply it to the agent. For the full per-iteration trace, set `print_detailed_results: true` in your optimization config file.

> **Long-running and expensive.** GEPA optimization makes many LLM calls and can take a long time. Do not run it unless the user explicitly asks for prompt optimization. When you do run it, iterate as far as possible with manual fixes first, then run a **single** final `eval optimize` — never loop on this command.

### Running the loop

Iterate stages 2 → 3 → 4 → 2 (with `synthesize`, re-run Stage 1 each pass, then `eval grade`). After each fix, run `agents-cli eval compare <prev_results>.json <new_results>.json` to confirm the target metric improved without regressing others. Expect 5–10+ iterations per case before it passes, which is normal. Only after a case passes should you expand coverage with more eval cases.

When doing 5+ iterations, maintain a task list of which cases are fixed, which are still failing, and what fixes you've tried. Prevents re-attempting the same fix.

**Hold cases back.** Keep a slice of cases out of the loop and grade them only when you think you're done — otherwise you can't tell a fix that generalizes from one fitted to the cases you iterated against.

### Shortcuts That Waste Time

Recognize these rationalizations and push back — they always cost more time than they save:

| Shortcut | Why it fails |
|----------|-------------|
| "I'll lower the bar so it passes" | Lowering the bar hides real failures. If the agent can't meet the bar, fix the agent, don't move the bar. |
| "This eval case is flaky, I'll skip it" | Flaky evals reveal non-determinism in your agent. Fix with `temperature=0`, rubric-based metrics, or more specific instructions — don't delete the signal. |
| "I just need to fix the eval dataset, not the agent" | If you're always adjusting expected outputs, your agent has a behavior problem. Fix the instructions or tool logic first. |
| "I'll iterate until every case I have passes" | Nothing is left to detect overfitting to your own cases. See *Hold cases back* above. |

## Choosing the Right Metrics

Pick built-in metrics by what you want to measure. Only `multi_turn_task_success`, `multi_turn_trajectory_quality`, and `multi_turn_tool_use_quality` accept multi-turn traces; every other built-in 400s on one. When no built-in fits, write a custom metric (see *Evaluation Configuration Schema* below).

| Goal | Recommended built-in metrics |
|------|------------------------------|
| **Did the agent achieve the user's goal?** (catch-all for multi-turn agents) | `multi_turn_task_success` |
| **Was the agent's reasoning path logical and efficient?** | `multi_turn_trajectory_quality` |
| **Quality of tool / function calling across turns** | `multi_turn_tool_use_quality` |
| **Final response quality** (no ground-truth reference needed) | `final_response_quality` |
| **Factual grounding** (catch hallucinated claims, e.g., RAG agents) | `hallucination`, or `grounding` when the case carries a `context` field |
| **Safety policy compliance** | `safety` |
| **Match against a golden answer** | `final_response_match` (needs `reference` on the case) |
| **Different pass/fail criteria per case** | Put them on the case as `rubric_groups` and grade with a managed rubric metric. See `references/dataset_schema.md` (*Per-Case Rubrics*). |
| **Domain-specific check no built-in covers** | Write a custom `LLMMetric` (LLM-judge) or `CodeExecutionMetric` (deterministic Python). See *Evaluation Configuration Schema* below. |

Run `agents-cli eval metric list` to see all available built-ins. For full metric definitions and rubric details, see the [Agent Platform metric docs](https://cloud.google.com/gemini-enterprise-agent-platform/optimize/evaluation/manage-metrics) and `references/metrics-guide.md`.

---

## What to fix when scores fail

After `agents-cli eval run` completes, inspect the latest `artifacts/grade_results/results_<timestamp>.json` (or open the `.html` file) for per-case scores and judge rationales, the input to every fix decision below.

| Failure | What to change |
|---------|---------------|
| `multi_turn_task_success` low | The agent isn't completing the user's goal — fix orchestration, missing tool calls, premature termination, or wrong tool selection |
| `multi_turn_trajectory_quality` low | The agent reaches the goal inefficiently or takes wrong steps — refine planning prompts, tighten instruction order, or remove redundant tool calls |
| `multi_turn_tool_use_quality` low | Fix tool descriptions, parameter docstrings, or agent instructions for tool selection |
| `final_response_quality` low | Read the auto-generated rubric verdicts; refine agent instructions to address the worst-scoring criterion (often clarity, completeness, or instruction-following) |
| `hallucination` low | Tighten agent instructions to stay grounded in tool output; verify the tool actually returned the data the agent claimed |
| `safety` low | Add safety guardrails to instructions; review the violating content category in the rubric verdict |
| Agent calls wrong tools | Fix tool descriptions, agent instructions, or `tool_config` |
| Agent calls extra tools | Add strict stop instructions, or switch to `multi_turn_tool_use_quality` |

After applying a fix, rerun `agents-cli eval run` and use `agents-cli eval compare <prev_results>.json <new_results>.json` to confirm the fix improved the target metric without regressing others.

---

## Eval Commands

`agents-cli eval <subcommand> --help` is the authoritative flag list; the examples below are the common invocations.

### `eval run` (default)

Runs the agent over the dataset and grades the traces in one command.

```bash
# Basic: dataset from tests/eval/datasets/, results to artifacts/grade_results/,
# metrics from tests/eval/eval_config.yaml
agents-cli eval run

# Advanced: pick the dataset, metrics, and output dir
agents-cli eval run --dataset tests/eval/datasets/custom.json --metrics final_response_quality,safety --output ./out/
```

### `eval generate`

Runs an agent over an evaluation dataset and writes traces to disk.

By default, runs the agent in a local HTTP server (launches the project's `fast_api_app.py` if it exists, or falls back to `adk api_server`) and sends each evaluation case over HTTP. You can generate traces from an already-running agent by passing its HTTP endpoint and app name to `--url` and `--app-name`.

```bash
# Basic — uses tests/eval/datasets/, writes to artifacts/traces/
agents-cli eval generate

# Advanced — custom dataset and output dir
agents-cli eval generate --dataset tests/eval/datasets/custom.json -o ./custom_traces/

# Against a deployed agent (or one you started manually)
agents-cli eval generate --url https://my-agent.run.app --app-name app
```

### `eval grade`

Scores traces (from `eval generate`, `eval dataset synthesize`, or hand-authored) against built-in or custom metrics. Writes timestamped `results_<YYYYMMDD_HHMMSS>.json` (consumed by `eval compare`) and `.html` (open in a browser) into the output dir, and prints a summary table to the console.

```bash
# Basic — defaults: traces from artifacts/traces/, results to artifacts/grade_results/,
# metrics from tests/eval/eval_config.yaml's metrics_to_run
agents-cli eval grade

# Advanced 1 — grade traces from a non-default location (the canonical
# pairing for `eval generate --output custom_traces/`)
agents-cli eval grade --traces custom_traces/

# Advanced 2: load metrics to run from a config file (YAML or JSON) on a specified trace file.
agents-cli eval grade --traces ./artifacts/traces/trace_1.json --config tests/eval/eval_config.yaml
```

See *Evaluation Configuration Schema* below for the config file format.

### `eval compare`

Diffs two `results_*.json` files from an eval run. Run it after a fix to confirm the target metric improved without regressing others.

```bash
agents-cli eval compare baseline.json candidate.json
```

### `eval dataset synthesize`

Generates user scenarios from your agent's tools and instructions, plays each against an LLM-backed user simulator, and writes graded-ready traces to `artifacts/traces/` (feed straight to `eval grade`, skip `eval generate`). Invocations, flags, and compatible metrics: `references/user-simulation.md`.

### Advanced commands

`eval analyze` (cluster failure modes), `eval optimize` (GEPA prompt tuning), and `eval submit` / `eval results` (managed cloud-side runs for CI or large datasets) are documented in `references/advanced-commands.md`.

---

## Evaluation Dataset Format

An `EvaluationDataset` is a JSON file with an `eval_cases` array. Cases come in two shapes depending on how they're used:

- **Inference input** (what you give to `eval generate`) — a user prompt or a partial conversation ending in a user prompt. The agent runs and produces traces.
- **Grading input** (what you give to `eval grade`) — a complete trace including the agent's responses and tool calls. Normally produced by `eval generate` or `eval dataset synthesize`; you don't write these by hand.

See `references/dataset_schema.md` for the full canonical schema, all field types, and common mistakes.

### Inference input format

Two shapes are supported.

**(a) Simple single-turn prompt** — what the scaffolded `tests/eval/datasets/basic-dataset.json` uses. The agent runs from scratch.

```json
{
  "eval_cases": [
    {
      "eval_case_id": "greeting",
      "prompt": {
        "role": "user",
        "parts": [{"text": "Hello, what can you help me with?"}]
      }
    }
  ]
}
```

**(b) Multi-turn continuation via `agent_data`** — a partial conversation whose last turn ends with a user message; the agent's next response is evaluated. See `references/dataset_schema.md` (*Multi-Turn / Multi-Agent Dataset*) for the JSON shape.

### Grading input format (traces)

A complete trace — agent responses plus `function_call` / `function_response` parts — normally produced by `eval generate` / `eval dataset synthesize` (you don't write these by hand). Authors are `"user"`, an agent ID from the `agents` map, or `"tool"`. See `references/dataset_schema.md` for the trace shape, multi-agent examples, and the full type reference.

---

## Evaluation Configuration Schema

`agents-cli eval run --config <path>` (and `eval grade --config <path>`) accepts a single configuration file in either **YAML** (`.yaml` / `.yml`) or **JSON** (`.json`). The file declares two parts:

- `metrics_to_run`: the **selection list** of metric names to execute on this run. A name resolves to a `custom_metrics` entry when one matches, otherwise to the built-in metric of that name.
- `custom_metrics` — a **definition pool** of custom metrics available to this project. Defining a metric here does **not** run it; it must also appear in `metrics_to_run` (or be passed via `--metrics name1,name2` on the CLI, which is equivalent to overriding `metrics_to_run` for that invocation).

**Minimal example (YAML preferred — human-readable, no JSON escaping for prompts and Python):**

```yaml
metrics_to_run:
  - multi_turn_task_success     # built-in
  - example_llm_metric          # selected from custom_metrics pool below
  - agent_turn_count            # selected from custom_metrics pool below

custom_metrics:
  - name: example_llm_metric
    prompt_template: |
      Rate the agent's response 1-5 for helpfulness and accuracy.
      Prompt: {prompt}
      Final response: {response}
      Full trace (for tool-call and reasoning context): {agent_data}
      Return JSON: {"score": <1|2|3|4|5>, "explanation": "<reason>"}

  - name: agent_turn_count
    custom_function: |
      def evaluate(instance):
          turns = (instance.get("agent_data") or {}).get("turns", [])
          return {'score': len(turns)}
```

JSON is also accepted (same field names, with `prompt_template` and `custom_function` as escaped strings) — but **always prefer YAML** for human-readable configs.

Dispatch by field: `custom_function` → Python metric; `prompt_template` → `LLMMetric` (LLM-as-judge); neither, on a built-in name → parameterizes that built-in (e.g. `metric_spec_parameters.rubric_group_key`). Field reference: `references/metrics-guide.md`.

**Agent trace field model.** For datasets produced by `agents-cli eval generate` (or `eval dataset synthesize`), each eval case exposes three standard fields to a metric:

- `{prompt}` — the user message (or first user turn).
- `{response}` — the agent's final text response, extracted from the last text-bearing event. In `custom_function` callbacks this is `instance['response']` with shape `{"role": "model", "parts": [{"text": "..."}]}`.
- `{agent_data}` — the full structured `turns`/`events` trace, useful when the judge needs to reason about tool calls or intermediate reasoning.

`reference`, `context`, and `rubric_groups` are yours to author on the case: `eval generate` carries them onto the trace but never invents them, so `{reference}` / `{context}` resolve only where you wrote them. `rubric_groups` is not a placeholder at all: managed rubric metrics read it off the case, and a `custom_function` sees `instance['rubric_groups']`. See `references/dataset_schema.md` (*Per-Case Rubrics*).

Code-based metrics default to **local in-process execution** (no GCP project or region required, but the `evaluate(instance)` function runs with the CLI's privileges). Set `execution: "remote"` on the metric to run it server-side in Vertex AI's `CodeExecutionMetric` sandbox instead — that path requires a configured GCP project + region.

---


## Common Gotchas

### Use Rubric-Based Tool Evaluation instead of Hardcoded Sequences

Evaluating agent tool usage using strict sequence matching is fragile because agents may call helper tools (like searches or geocoding) in different orders or perform extra proactive steps.

Instead, use **`multi_turn_tool_use_quality`** / **`multi_turn_trajectory_quality`**. These metrics automatically generate content-based and intent-based adaptive rubrics, assessing technical correctness and technical sequence logic semantically using an LLM judge rather than forcing a rigid match.

### App name must match directory name

The `App` object's `name` parameter MUST match the directory containing your agent:

```python
# CORRECT - matches the "app" directory
app = App(root_agent=root_agent, name="app")

# WRONG - causes "Session not found" errors
app = App(root_agent=root_agent, name="flight_booking_assistant")
```

### Vertex eval region

`eval run`, `eval grade`, and `eval submit` **default to the `global` endpoint**. They don't inherit the manifest `region` (the eval services support only a subset of regions), and `eval analyze` is `global`-only. Override these per run with `--region <REGION>` (e.g. data residency); the service rejects an unsupported one:

```
400 FAILED_PRECONDITION: Unsupported region for Vertex Evaluation Service: <region>
```

`eval generate` (without the `--url` flag) and `eval dataset synthesize` run your agent locally, so they honor the agent's own `.env` — notably `GOOGLE_CLOUD_LOCATION`, which selects the model endpoint **when the agent uses Vertex AI** (`GOOGLE_GENAI_USE_VERTEXAI=true`); it's unused with a `GEMINI_API_KEY` (AI Studio). They take **no** `--region` and never override your `.env` with the manifest `region`; change the model region by editing `.env`. One caveat for `synthesize`: its scenario-generation step is a **server-side** eval call at `GOOGLE_CLOUD_LOCATION`, so keep that an eval-supported region (`global` by default) even though the agent itself could run elsewhere.

**No eval region fits your data-residency rules?** Fall back to **local custom metrics** — a `custom_metrics` entry with a `custom_function` (`execution: local`, the default) grades in-process with no GCP region required. You lose the managed built-in metrics, but your `custom_function` can still call an LLM judge in a compliant region itself — so LLM-as-judge grading stays available anywhere.

### The `before_agent_callback` Pattern (State Initialization)

Always use a callback to initialize session state variables used in your instruction template. This prevents `KeyError` crashes on the first turn:

```python
async def initialize_state(callback_context: CallbackContext) -> None:
    state = callback_context.state
    if "user_preferences" not in state:
        state["user_preferences"] = {}

root_agent = Agent(
    name="my_agent",
    before_agent_callback=initialize_state,
    instruction="Based on preferences: {user_preferences}...",
)
```

### Model thinking mode may bypass tools

Models with "thinking" enabled may skip tool calls. Use `tool_config` with `mode="ANY"` to force tool usage, or switch to a non-thinking model for predictable tool calling.

---

## Common Eval Failure Causes

| Symptom | Cause | Fix |
|---------|-------|-----|
| Score fluctuates between runs | Non-deterministic model | Set `temperature=0` or use rubric-based eval with multiple samples |
| LLM judge ignores image/audio in eval | `get_text_from_content()` skips non-text parts | Use custom metric with vision-capable judge (see `references/multimodal-eval.md`) |

---

## Proving Your Work

Don't assert that eval passes — show the evidence. Concrete output prevents false confidence and catches issues early.

- **After running eval:** Paste the scores table output so the user can see exactly what passed and failed.
- **After fixing a failure:** Show before/after scores for the specific case you fixed, and confirm no other cases regressed.
- **Before deploy:** Rerun `agents-cli eval run` and show the scores for every case, not just the one you fixed. `eval run` exits 0 whatever the scores are, so the numbers you paste are the gate, not the exit code.

---

## Related Skills

- `/google-agents-cli-workflow` — Development workflow and the spec-driven build-evaluate-deploy lifecycle
- `/google-agents-cli-adk-code` — ADK Python API quick reference for writing agent code
- `/google-agents-cli-scaffold` — Project creation and enhancement with `agents-cli scaffold create` / `scaffold enhance`
- `/google-agents-cli-deploy` — Deployment targets, CI/CD pipelines, and production workflows
- `/google-agents-cli-observability` — Cloud Trace, logging, and monitoring for debugging agent behavior