google-agents-cli-observability

google/agents-cli/google-agents-cli-observability

This skill guides users in setting up observability for deployed ADK (Agent Development Kit) agents, covering Cloud Trace, prompt-response logging, BigQuery Agent Analytics, and third-party integrations such as AgentOps, Phoenix, and MLflow. It provides a tiered approach to observability, from always-on Cloud Trace to optional BigQuery analytics and external platforms, along with troubleshooting guidance. The skill also explains infrastructure provisioning requirements (e.g., running `agents-cli infra single-project`) and configuration details for different deployment types (e.g., Agent Runtime, Cloud Run). It helps users choose the right observability tier for their needs and provides verification commands and pointers to reference documentation.

安裝量 · 1,066查看來源

Installation

npx skills add https://github.com/google/agents-cli --skill google-agents-cli-observability

技能檔案

SKILL.md

最近同步 · 2026年8月29日

references/bigquery-agent-analytics.md›

# BigQuery Agent Analytics Plugin

> **Opt-in.** Enable with `--bq-analytics` at scaffold time, or add manually to `app/agent.py`.

An optional plugin that logs structured agent events directly to BigQuery via the Storage Write API. Enables:

- **Conversational analytics** — session flows, user interaction patterns
- **LLM-as-judge evals** — structured data for evaluation pipelines
- **Custom dashboards** — Looker Studio integration
- **Tool provenance tracking** — LOCAL, MCP, SUB_AGENT, A2A, TRANSFER_AGENT

## Enabling

| Method | How |
|--------|-----|
| **At scaffold time** | `agents-cli scaffold create <project-name> --bq-analytics` |
| **Post-scaffold** | Add the plugin manually to `app/agent.py` (see [ADK docs](https://adk.dev/integrations/bigquery-agent-analytics/index.md)) |

Infrastructure (BigQuery dataset, GCS offloading) is provisioned automatically by Terraform when enabled at scaffold time.

## Key Features

- Auto-schema upgrade (new fields added without migration)
- GCS offloading for multimodal content (images, audio)
- Distributed tracing via OpenTelemetry span context
- SQL-queryable event log for all agent interactions

For full schema, SQL query examples, and Looker Studio setup, fetch `https://adk.dev/integrations/bigquery-agent-analytics/index.md`.

references/cloud-trace-and-logging.md›

# Cloud Trace & Prompt-Response Logging (Scaffolded Projects)

> **Assumes `/google-agents-cli-scaffold` scaffolding.** Observability infrastructure is provisioned by Terraform in scaffolded projects.

## Cloud Trace

Always-on distributed tracing, exporting spans/logs to Cloud Trace and Cloud Logging via `get_fast_api_app(otel_to_cloud=True)`. For **Agent Runtime** it's gated on `GOOGLE_CLOUD_AGENT_ENGINE_ENABLE_TELEMETRY` (set by deploy), and traces also appear in the Agent Engine console. Content env vars are declared statically (Terraform `service.tf` for deployed, `.env` for local). Tracks requests through LLM calls and tool executions with latency analysis and error visibility.

View traces: **Cloud Console → Trace → Trace explorer**

No configuration required. Works in local dev (`agents-cli playground`) and all deployed environments.

## Prompt-Response Logging Infrastructure

All provisioned automatically by `deployment/terraform/single-project/telemetry.tf` (and the `cicd/` variant):

- **Log sinks** — Route GenAI inference logs directly to BigQuery (partitioned tables)
- **BigQuery dataset** — Telemetry dataset with external tables over GCS data and pre-created log export table
- **Pre-created log export table** — Cloud Logging BQ export schema (labels flattened: dots become underscores). Cloud Logging names the sink table after the log id, so it varies by deployment target: `gen_ai_client_inference_operation_details` (Cloud Run / GKE) or `aiplatform_googleapis_com_reasoning_engine_stdout` (Agent Runtime, where GenAI logs arrive via stdout)
- **GCS logs bucket** — Stores completions as NDJSON
- **BigQuery connection** — Service account for GCS access from BigQuery
- **Completions view** — Joins BQ log export data with GCS-stored prompt/response data

Check `deployment/terraform/single-project/telemetry.tf` for exact configuration. IAM bindings grant log sink service accounts `roles/bigquery.dataEditor` on the telemetry dataset.

**Collecting user feedback?** The same infrastructure supports a feedback mechanism (endpoint + structured logging → log sink → BigQuery). See `references/feedback-mechanism.md`.

## Environment Variables

Set automatically by Terraform on the deployed service. The `OTEL_INSTRUMENTATION_GENAI_*` content-capture and upload variables (capture modes, `OTEL_SEMCONV_STABILITY_OPT_IN`, completion hook / upload) are documented in [`opentelemetry-util-genai`](https://github.com/open-telemetry/opentelemetry-python-genai/tree/main/util/opentelemetry-util-genai) and the [OpenTelemetry GenAI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/).

| Variable | Purpose |
|----------|---------|
| `LOGS_BUCKET_NAME` | GCS bucket for completions and logs. Required to enable prompt-response logging |
| `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT` | Controls content capture for the traces/events tier only (`NO_CONTENT`/`EVENT_ONLY`/`SPAN_ONLY`/`SPAN_AND_EVENT`; `true`/`false` invalid) |
| `ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS` | Keeps message content out of trace spans; Terraform sets `false` (ADK defaults to `true`) |
| `BQ_ANALYTICS_DATASET_ID` | BigQuery dataset for telemetry (only when scaffolded with `--bq-analytics`) |
| `BQ_ANALYTICS_CONNECTION_ID` | BigQuery connection for GCS access (only when scaffolded with `--bq-analytics`) |
| `BQ_ANALYTICS_GCS_BUCKET` | GCS bucket for BigQuery Analytics multimodal offloading (only when scaffolded with `--bq-analytics`) |
| `OTEL_INSTRUMENTATION_GENAI_COMPLETION_HOOK` | Set to `upload` to export full completions to GCS (the prompt-response logging feature) |
| `OTEL_INSTRUMENTATION_GENAI_UPLOAD_BASE_PATH` | GCS path for uploaded completions (e.g. `gs://<bucket>/completions`) |
| `OTEL_INSTRUMENTATION_GENAI_UPLOAD_FORMAT` | Upload format for completions (e.g. `jsonl`) |

## Enabling / Disabling

### Enable Locally

Telemetry config is env-var driven, so set the same vars Terraform sets for deployed agents before running `agents-cli playground`:

```bash
export LOGS_BUCKET_NAME="your-bucket-name"                       # bare name, no gs://
export OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT="NO_CONTENT"  # or EVENT_ONLY (content in Cloud Logging events)
export OTEL_INSTRUMENTATION_GENAI_COMPLETION_HOOK="upload"
export OTEL_INSTRUMENTATION_GENAI_UPLOAD_BASE_PATH="gs://your-bucket-name/completions"
export OTEL_INSTRUMENTATION_GENAI_UPLOAD_FORMAT="jsonl"
export OTEL_SEMCONV_STABILITY_OPT_IN="gen_ai_latest_experimental"
```

### Disable in Deployed Environments

Content in traces/events is already off by default (`NO_CONTENT`) — note `true`/`false` are **not** valid values under experimental semconv (they fall back to `NO_CONTENT`). To turn off prompt-response logging to GCS/BigQuery entirely, remove the upload block (`OTEL_INSTRUMENTATION_GENAI_COMPLETION_HOOK`, `OTEL_INSTRUMENTATION_GENAI_UPLOAD_BASE_PATH`, `LOGS_BUCKET_NAME`) from `deployment/terraform/single-project/service.tf` (or the `cicd/` variant) and re-apply Terraform.

## BigQuery Dataset Naming Convention

BigQuery dataset names **cannot contain hyphens**. Terraform automatically converts hyphens to underscores when creating dataset names from your project name:

- Project name `my-agent` → BQ dataset `my_agent_telemetry`

One dataset is created:
- **`{name}_telemetry`** — Contains external tables over GCS completions data (NDJSON), the pre-created log export table (`gen_ai_client_inference_operation_details` on Cloud Run / GKE, `aiplatform_googleapis_com_reasoning_engine_stdout` on Agent Runtime), and the `completions_view`

To discover the actual dataset name in your project:
```bash
bq ls --project_id=${PROJECT_ID}
```

## Verifying Telemetry

After deploying, verify prompt-response logging is working:

```bash
PROJECT_ID="your-dev-project-id"
PROJECT_NAME="your-app-name"  # The agents-cli project name (not the GCP project ID)

# Check GCS data
gsutil ls gs://${PROJECT_ID}-${PROJECT_NAME}-logs/completions/

# Check BigQuery log export table (logs arrive via sink, may take a few minutes).
# Table name varies by target: gen_ai_client_inference_operation_details on
# Cloud Run / GKE, aiplatform_googleapis_com_reasoning_engine_stdout on Agent Runtime.
bq query --use_legacy_sql=false \
  "SELECT COUNT(*) FROM \`${PROJECT_ID}.${PROJECT_NAME//-/_}_telemetry.gen_ai_client_inference_operation_details\`"

# Query completions external table
bq query --use_legacy_sql=false \
  "SELECT * FROM \`${PROJECT_ID}.${PROJECT_NAME//-/_}_telemetry.completions\` LIMIT 10"

# Query the completions view (joins log export with GCS data)
bq query --use_legacy_sql=false \
  "SELECT * FROM \`${PROJECT_ID}.${PROJECT_NAME//-/_}_telemetry.completions_view\` LIMIT 10"
```

If data is not appearing: check `LOGS_BUCKET_NAME` is set, verify SA has `storage.objectCreator` on the bucket, check application logs for telemetry setup warnings. Log export to BigQuery may take a few minutes to propagate.

references/feedback-mechanism.md›

# Feedback Mechanism (Scaffolded Projects)

> **Assumes `/google-agents-cli-scaffold` scaffolding.** Reuses the same telemetry infrastructure documented in `cloud-trace-and-logging.md`.

To collect end-user feedback (ratings, thumbs up/down, free-text) and land it in BigQuery for analysis, reuse the same pattern as GenAI logs: **structured log → log sink → BigQuery**. There are three components.

## 1. Request model

A Pydantic model with a fixed discriminator field so the log sink can filter on it. Put it wherever your app keeps request/response models (e.g. `app/app_utils/typing.py`):

```python
import uuid
from typing import Literal

from pydantic import BaseModel, Field


class Feedback(BaseModel):
    """Represents feedback for a conversation."""

    score: int | float
    text: str | None = ""
    log_type: Literal["feedback"] = "feedback"
    service_name: Literal["<project-name>"] = "<project-name>"
    user_id: str = Field(default_factory=lambda: str(uuid.uuid4()))
    session_id: str = Field(default_factory=lambda: str(uuid.uuid4()))
```

The `log_type` and `service_name` fields are what the log sink filters on — keep them stable.

> **User text is retained.** `text` is free-form user input and lands in Cloud Logging and BigQuery — redact or omit it if you can't retain PII, keep `user_id`/`session_id` opaque (the defaults are random UUIDs), and set a table expiration on the telemetry dataset if you do.

## 2. FastAPI endpoint

An endpoint in `app/fast_api_app.py` that writes the payload as a **structured** log entry (so it lands in `jsonPayload`, not a plain text message). Create a Cloud Logging client once at module scope:

```python
from google.cloud import logging as google_cloud_logging

logging_client = google_cloud_logging.Client()
logger = logging_client.logger(__name__)


@app.post("/feedback")
def collect_feedback(feedback: Feedback) -> dict[str, str]:
    """Collect and log feedback."""
    logger.log_struct(feedback.model_dump(), severity="INFO")
    return {"status": "success"}
```

`severity="INFO"` keeps the entries out of error alerting. `log_struct` writes each field into `jsonPayload`, which the sink filter matches against.

## 3. Terraform log sink → BigQuery

A log sink in `deployment/terraform/single-project/telemetry.tf` (and the `cicd/` variant) that routes feedback entries to the telemetry BigQuery dataset, plus an IAM binding granting the sink's `writer_identity` write access:

```hcl
resource "google_logging_project_sink" "feedback_logs_to_bq" {
  name                   = "${var.project_name}-feedback"
  project                = var.project_id
  destination            = "bigquery.googleapis.com/projects/${var.project_id}/datasets/${google_bigquery_dataset.telemetry_dataset.dataset_id}"
  filter                 = "jsonPayload.log_type=\"feedback\" jsonPayload.service_name=\"${var.project_name}\""
  unique_writer_identity = true

  bigquery_options {
    use_partitioned_tables = true
  }

  depends_on = [google_bigquery_dataset.telemetry_dataset]
}

resource "google_bigquery_dataset_iam_member" "feedback_logs_bq_writer" {
  project    = var.project_id
  dataset_id = google_bigquery_dataset.telemetry_dataset.dataset_id
  role       = "roles/bigquery.dataEditor"
  member     = google_logging_project_sink.feedback_logs_to_bq.writer_identity
}
```

For the **cicd** variant, add `for_each = local.deploy_project_ids` and index the referenced resources with `[each.key]` / `[each.value]`, matching the other sinks in that file.

## Verify

On first write the sink auto-creates a date-partitioned table (named after the log) in the telemetry dataset. After POSTing a feedback payload, confirm the log entry:

```bash
gcloud logging read 'jsonPayload.log_type="feedback"' --limit 5 --project PROJECT_ID
```

Then query the exported table in the `<project_name>_telemetry` BigQuery dataset (a few minutes after the first write) to confirm the sink is delivering rows.

SKILL.md›

---
name: google-agents-cli-observability
description: >
  This skill should be used when the user wants to "set up tracing",
  "monitor my ADK agent", "configure logging", "add observability",
  "debug production traffic", or needs guidance on monitoring deployed
  ADK (Agent Development Kit) agents.
  Covers Cloud Trace, prompt-response logging, BigQuery Agent Analytics,
  third-party integrations (AgentOps, Phoenix, MLflow, etc.), and troubleshooting.
  Part of the Google ADK (Agent Development Kit) skills suite.
  Do NOT use for deployment setup (use google-agents-cli-deploy) or
  API code patterns (use google-agents-cli-adk-code).
metadata:
  author: Google
  license: Apache-2.0
  version: 1.4.2
  requires:
    bins:
      - agents-cli
    install: "uv tool install google-agents-cli"
---

# ADK Observability Guide

> **Cloud Trace** works out of the box — no infrastructure needed. **Prompt-response logging** and **BigQuery Agent Analytics** require Terraform-provisioned infrastructure (service account, GCS bucket, BigQuery dataset). Run `agents-cli infra single-project --project PROJECT_ID` to provision these resources. See `references/cloud-trace-and-logging.md` for details, env vars, and verification commands. If your project isn't scaffolded yet, see `/google-agents-cli-scaffold` first.

### Order of operations for `agent_runtime` deployments

For `deployment_target = agent_runtime`, run `agents-cli infra single-project` **before** the first `agents-cli deploy`. The Terraform module owns the entire Reasoning Engine resource (service account, deployment spec, env vars), so applying it after an SDK-based deploy creates a state mismatch Terraform can't reconcile without taking ownership of the whole resource.

Already ran `agents-cli deploy`? Two options:

1. **Switch to Terraform-managed** — delete the SDK-deployed Reasoning Engine, then run `agents-cli infra single-project` and `agents-cli deploy` (sessions and in-flight state are lost).
2. **Keep the SDK-deployed instance** — skip `infra single-project` and set the observability env vars by re-running `agents-cli deploy --update-env-vars "KEY=VALUE,..."`; deploy matches the existing Reasoning Engine by display name and updates it in place, preserving env vars set outside the deploy. You must also grant its service account the telemetry IAM roles the Terraform module would otherwise provision: `roles/storage.admin` (write completions to the logs bucket), `roles/logging.logWriter`, `roles/cloudtrace.agent`, plus `roles/bigquery.dataOwner` + `roles/bigquery.jobUser` when scaffolded with `--bq-analytics`. The full set lives in `deployment/terraform/single-project/iam.tf` (from `app_sa_roles`) and `telemetry.tf`. Terraform-managed env vars aren't available in this mode.

### Reference Files

| File | Contents |
|------|----------|
| `references/cloud-trace-and-logging.md` | Scaffolded project details — Terraform-provisioned resources, environment variables, verification commands, enabling/disabling locally |
| `references/bigquery-agent-analytics.md` | BQ Agent Analytics plugin — enabling, key features, GCS offloading, tool provenance |
| `references/feedback-mechanism.md` | Adding a user-feedback endpoint — request model, structured logging, log sink → BigQuery |

---

## Observability Tiers

Choose the right level of observability based on your needs:

| Tier | What It Does | Scope | Default State | Best For |
|------|-------------|-------|---------------|----------|
| **Cloud Trace** | Distributed tracing — execution flow, latency, errors via OpenTelemetry spans | All templates, all environments | Always enabled | Debugging latency, understanding agent execution flow |
| **Prompt-Response Logging** | GenAI interactions exported to GCS, BigQuery, and Cloud Logging | ADK agents only | Disabled locally, enabled when deployed | Auditing LLM interactions, compliance |
| **BigQuery Agent Analytics** | Structured agent events (LLM calls, tool use, outcomes) to BigQuery | ADK agents with plugin enabled | Opt-in (`--bq-analytics` at scaffold time) | Conversational analytics, custom dashboards, LLM-as-judge evals |
| **Third-Party Integrations** | External observability platforms (AgentOps, Phoenix, MLflow, etc.) | Any ADK agent | Opt-in, per-provider setup | Team collaboration, specialized visualization, prompt management |

**Ask the user** which tier(s) they need — they can be combined. Cloud Trace is always on; the others are additive.

---

## Cloud Trace

ADK uses OpenTelemetry to emit distributed traces. Every agent invocation produces spans that track the full execution flow.

### Span Hierarchy

```
invoke_workflow (top-level run)
  └── invoke_agent (one per agent in the chain)
        ├── call_llm (model request)
        │     └── generate_content (underlying GenAI model call)
        └── execute_tool (tool execution)
```

### Setup by Deployment Type

| Deployment | Setup |
|-----------|-------|
| **Agent Runtime** | Automatic — `get_fast_api_app(otel_to_cloud=True)`, gated on `GOOGLE_CLOUD_AGENT_ENGINE_ENABLE_TELEMETRY` (set by deploy); exports to Cloud Trace/Logging + Agent Engine console |
| **Cloud Run / GKE (scaffolded)** | Automatic — `get_fast_api_app(otel_to_cloud=True)` exports to Cloud Trace/Logging |
| **Cloud Run / GKE (manual)** | Configure OpenTelemetry exporter in your app |
| **Local dev** | Works with `agents-cli playground`; traces visible in Cloud Console |

View traces: **Cloud Console → Trace → Trace explorer**

For detailed setup instructions (Agent Runtime CLI/SDK, Cloud Run, custom deployments), fetch `https://adk.dev/integrations/cloud-trace/index.md`.

---

## Prompt-Response Logging

Captures GenAI interactions and exports to GCS (JSONL) and BigQuery (via log sinks + external tables). Content is governed by **two independent tiers**; the net Terraform-deploy default is **full content in GCS/BigQuery, none in traces**:

| Tier | Captures | Controlled by | Default (Terraform deploy) |
|------|----------|---------------|----------------------------|
| **GCS/BigQuery completions** | Full prompts/responses (the prompt-response logging feature) | `OTEL_INSTRUMENTATION_GENAI_COMPLETION_HOOK=upload` + `LOGS_BUCKET_NAME` | **On** — full content |
| **Trace spans / Cloud Logging events** | Span/event content | `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT` + `ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS=false` | **Off** — `NO_CONTENT` |

The tiers are independent: GCS/BigQuery uploads capture full content whenever their upload vars are set and do **not** honor `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT`, which governs the traces/events tier only. Its valid (experimental-semconv) values:

- `NO_CONTENT` — no content in spans/events (scaffolded default)
- `EVENT_ONLY` — content in Cloud Logging events
- `SPAN_ONLY` / `SPAN_AND_EVENT` — content in trace spans
- `true` / `false` — **invalid**; fall back to `NO_CONTENT`

For the full mechanics (semconv opt-in, declarative Terraform config, env-var table, enabling/disabling, verification commands), see `references/cloud-trace-and-logging.md`. For ADK logging docs (log levels, configuration, debugging), fetch `https://adk.dev/observability/logging/index.md`.

---

## BigQuery Agent Analytics Plugin

Optional plugin that logs structured agent events to BigQuery. Enable with `--bq-analytics` at scaffold time. See `references/bigquery-agent-analytics.md` for details.

---

## Third-Party Integrations

ADK supports many third-party observability platforms (via OpenTelemetry or custom instrumentation). The table below covers common ones; the full list is larger (see the pointer below it).

| Platform | Key Differentiator | Setup Complexity | Self-Hosted Option |
|----------|-------------------|-----------------|-------------------|
| **AgentOps** | Session replays, 2-line setup, replaces native telemetry | Minimal | No (SaaS) |
| **Arize AX** | Commercial platform, production monitoring, evaluation dashboards | Low | No (SaaS) |
| **Phoenix** | Open-source, custom evaluators, experiment testing | Low | Yes |
| **MLflow** | OTel traces to MLflow Tracking Server, span tree visualization | Medium (needs SQL backend) | Yes |
| **Monocle** | 1-call setup, VS Code Gantt chart visualizer | Minimal | Yes (local files) |
| **Weave** | W&B platform, team collaboration, timeline views | Low | No (SaaS) |
| **Freeplay** | Prompt management + evals + observability in one platform | Low | No (SaaS) |

**Ask the user** which platform they prefer — present the trade-offs and let them choose. Fetch a platform's setup page at `https://adk.dev/integrations/<slug>/index.md` (slugs for the table above: `agentops`, `arize-ax`, `phoenix`, `mlflow-tracing`, `monocle`, `weave`, `freeplay`). ADK has more observability integrations (Datadog, Galileo, LangWatch, Latitude, Future AGI, Respan, Zespan, …) — browse the complete, current list at `https://adk.dev/integrations/` (observability topic).

---

## Troubleshooting

| Issue | Solution |
|-------|----------|
| No traces in Cloud Trace | Verify `fast_api_app.py` uses `get_fast_api_app(otel_to_cloud=True)` (Agent Runtime gates it on `GOOGLE_CLOUD_AGENT_ENGINE_ENABLE_TELEMETRY`) and the SA has the `cloudtrace.agent` role |
| Prompt-response data not appearing | Check `LOGS_BUCKET_NAME` is set; verify SA has `storage.objectCreator` on the bucket; check app logs for telemetry setup warnings |
| Content in traces/events (unwanted) | `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=NO_CONTENT` keeps content out of spans/events. NOTE: GCS/BigQuery completions still capture full content — to stop that, remove `LOGS_BUCKET_NAME`/`OTEL_INSTRUMENTATION_GENAI_COMPLETION_HOOK` (drop the upload block in `service.tf`) |
| BigQuery Analytics not logging | Verify plugin is configured in `app/agent.py`; check `BQ_ANALYTICS_DATASET_ID` env var is set |
| Third-party integration not capturing spans | Check provider-specific env vars (API keys, endpoints); some providers (AgentOps) replace native telemetry |
| Traces missing tool spans | Tool execution spans appear under `execute_tool` — check trace explorer filters |
| High telemetry costs | Switch to `NO_CONTENT` mode; reduce BigQuery retention; disable unused tiers |

---

## Deep Dive: ADK Docs (WebFetch URLs)

For detailed documentation beyond what this skill covers, fetch these pages:

| Topic | URL |
|-------|-----|
| Observability overview | `https://adk.dev/observability/index.md` |
| Agent activity logging | `https://adk.dev/observability/logging/index.md` |
| Cloud Trace integration | `https://adk.dev/integrations/cloud-trace/index.md` |
| BigQuery Agent Analytics | `https://adk.dev/integrations/bigquery-agent-analytics/index.md` |

---

## Related Skills

- `/google-agents-cli-deploy` — Deployment targets, CI/CD pipelines, and production workflows
- `/google-agents-cli-workflow` — Development workflow, coding guidelines, and operational rules
- `/google-agents-cli-adk-code` — ADK Python API quick reference for writing agent code