Export traces and metrics

The daemon can export traces and metrics over OTLP to any OpenTelemetry collector or hosted backend. Export is off by default. When it is on, every exported value passes a sanitizing exporter that keeps IDs, fixed names, and counts, and never prompt text, tool payloads, paths, or credentials.

Configuration lives in the global config.toml under [observability.otel]. Every key is restart-required, so restart the daemon after a change.

Try it with a local stack

This stack accepts both signals: an OpenTelemetry Collector (OTLP on 4317/4318) forwards traces to Jaeger (UI on 16686) and metrics to Prometheus (UI on 9090).

~/otel-local/compose.yaml:

services:
  collector:
    image: otel/opentelemetry-collector-contrib:0.138.0
    command: ["--config=/etc/otelcol/config.yaml"]
    volumes:
      - ./collector.yaml:/etc/otelcol/config.yaml:ro
    ports:
      - "4317:4317" # OTLP gRPC
      - "4318:4318" # OTLP HTTP
  jaeger:
    image: jaegertracing/jaeger:2.11.0
    ports:
      - "16686:16686"
  prometheus:
    image: prom/prometheus:v3.7.3
    command: ["--config.file=/etc/prometheus/prometheus.yaml"]
    volumes:
      - ./prometheus.yaml:/etc/prometheus/prometheus.yaml:ro
    ports:
      - "9090:9090"

~/otel-local/collector.yaml:

receivers:
  otlp:
    protocols:
      grpc: { endpoint: 0.0.0.0:4317 }
      http: { endpoint: 0.0.0.0:4318 }
exporters:
  otlp/jaeger:
    endpoint: jaeger:4317
    tls: { insecure: true }
  prometheus:
    endpoint: 0.0.0.0:8889
service:
  pipelines:
    traces: { receivers: [otlp], exporters: [otlp/jaeger] }
    metrics: { receivers: [otlp], exporters: [prometheus] }

~/otel-local/prometheus.yaml:

scrape_configs:
  - job_name: compozy-collector
    scrape_interval: 15s
    static_configs:
      - targets: ["collector:8889"]

Start the stack, enable export, and restart the daemon:

$ cd ~/otel-local && docker compose up -d

$ compozy config set observability.otel.enabled true
Config
======
Path:               observability.otel.enabled
Value:              true
Scope:              user
Target:             /home/ana/.compozy/config.toml
Redacted:           false
Lifecycle:          restart-required
Applied:            false
Next Action:        restart-daemon
Apply Record:       cfgapp-2f7faa2253b9fd65
Active Generation:  0
Restart Required:   true
Restart Scope:      daemon
Status:             --
Winning Layer:      --

$ compozy daemon stop && compozy daemon start
$ compozy session prompt sess-9c1e "list my open sessions"
$ compozy status
Runtime
=======
Status:             running
...
Config:             ok
OTel:               ok · http/protobuf → localhost:4318 · traces ok, metrics ok · sample 1.0
Log Tail:           available

Open Jaeger at localhost:16686, pick service compozy-daemon and operation session.turn. The trace shows POST /api/workspaces/:workspace_id/sessions/:session_id/prompt → session.turn → acp.process.spawn on a cold session and acp session/prompt, with mcp.hosted.call → tool.execute (compozy.tool = compozy__session_list) nested inside session.turn.

Open Prometheus at localhost:9090 and query:

histogram_quantile(0.95, sum by (le) (rate(compozy_session_turn_duration_seconds_bucket[5m])))

Metrics show pending in compozy status until the first push, which happens every 60 seconds by default. The query returns NaN after a single turn, because rate() needs an increase between two samples; send a second prompt within the 5-minute window to get a value.

[observability] enabled must stay true; with it off, status reports inactive (observability.enabled = false) and nothing is exported.

For a fuller setup, the repository ships a Grafana OSS stack in infra/observability: Tempo, Loki, Prometheus and Grafana, with a provisioned CompozyOS daemon dashboard and cross-linked data sources. Start it with docker compose up -d in that directory and point observability.otel.endpoint at http://localhost:4318.

Configure export

[observability]
enabled = true                       # existing; OTel export requires it

[observability.otel]
enabled = false                      # master switch for OTLP export
protocol = "http/protobuf"           # "http/protobuf" | "grpc"
endpoint = ""                        # "" → OTEL_EXPORTER_OTLP_ENDPOINT → http://localhost:4318 (grpc: http://localhost:4317)
traces = true                        # export spans when enabled
metrics = true                       # export metrics when enabled
sample_ratio = 1.0                   # 0.0–1.0, new traces and unsampled parents
metric_interval = "60s"              # minimum "5s"
service_name = "compozy-daemon"      # service.name; letters, digits, . _ - (max 64)
slow_db_write_ms = 50                # write-span threshold; 0 = no write spans
forward_to_providers = false         # give provider processes OTEL_* + TRACEPARENT (operator policy)

[observability.otel.headers]         # values must be env: or vault:observability/ refs
# "api-key" = "vault:observability/newrelic-license"
# "Authorization" = "env:GRAFANA_OTLP_AUTH"

Scalar keys are settable with compozy config set; edit headers in the file.

$ compozy config set observability.otel.protocol grpc
$ compozy config set observability.otel.endpoint http://collector.local:4317
$ compozy config set observability.otel.sample_ratio 0.25
$ compozy config get observability.otel.enabled
observability.otel.enabled: true

All keys are global-only. A workspace config.toml that sets [observability.otel] fails to load, workspace-scope writes are rejected, and profile overlays deny the observability block.

Environment variables

Standard OpenTelemetry variables fill only the values config.toml does not set. A value written in the file wins even when it equals the default; an empty endpoint counts as not set.

Their values are checked only while export can run ([observability] enabled, observability.otel.enabled, and at least one of traces or metrics), so variables set for other tools never block a daemon that exports nothing. Only the daemon's process environment counts; a workspace .env file never sets them. compozy config get resolves them the same way, from the environment of the shell that runs it. Invalid environment endpoints are redacted in config inspection even while export is inactive; credentials belong in header refs, never in endpoint URLs.

Header secrets

Hosted backends authenticate with a header. Header values must be env: or vault:observability/ refs; literal values are rejected. Store the secret in the vault, then reference it:

$ printf '%s' "$NEW_RELIC_LICENSE_KEY" | compozy vault put vault:observability/newrelic-license --kind token --value-stdin
$ compozy config edit   # add the [observability.otel.headers] entry

[observability.otel.headers]
"api-key" = "vault:observability/newrelic-license"

The daemon resolves refs at start. compozy config get shows the ref, never the resolved value. Short header values, such as a tenant ID, are masked when they exactly match an exported string; they do not enter process-wide substring redaction or rewrite unrelated saved session content.

Agents

Agents can read the keys but not change them. compozy__config_set refuses every observability.otel.* key with config_trust_root_forbidden: telemetry settings decide what leaves the machine and where, so only the operator sets them with compozy config set or compozy config edit. compozy__config_get reads one key per call. It shows effective values, including the daemon's OTEL_* environment fallback where the file does not set a value. Header values read as [REDACTED] because the daemon masks every env:/vault: ref in agent-facing output. The operator reads the ref with compozy config get.

Read export status

compozy status shows one OTel line in the Runtime section:

Export reads disabled when observability.otel.enabled is false, when OTEL_SDK_DISABLED=true, or when both traces and metrics are false.

compozy status --json and GET /api/status (HTTP and UDS) carry the same data in otel:

{
  "otel": {
    "state": "ok",
    "protocol": "grpc",
    "endpoint_host": "collector.local:4317",
    "sample_ratio": 0.25,
    "forward_to_providers": false,
    "traces": {
      "enabled": true,
      "last_export_at": "2026-11-07T14:02:11Z",
      "failed_exports": 0
    },
    "metrics": {
      "enabled": true,
      "last_export_at": "2026-11-07T14:02:00Z",
      "failed_exports": 0
    },
    "dropped_spans": 0
  }
}

A degraded signal reports its last error class and failure count:

"metrics": { "enabled": true, "last_export_at": "2026-11-07T13:20:00Z", "last_error_class": "http_401", "failed_exports": 12 }

A daemon older than OTLP export omits otel; treat a missing object as unknown. Export failures never fail a command, and compozy status exits 0 whenever the daemon answers.

Every HTTP route accepts a W3C traceparent header, so an external caller can join its own trace. Invalid values are ignored and the request is served normally; tracestate is ignored.

What is exported

Everything below passes the daemon's sanitizing exporter. The resource carries service.name (config), service.version (daemon version), service.instance.id (daemon boot ID), os.type, and host.arch. Span status descriptions are always empty; failures carry error.type.

Spans

Metrics

Prometheus renames dots to underscores and adds the unit, so compozy.session.turn.duration becomes compozy_session_turn_duration_seconds.

Value rules

• Agent, provider, model, and tool values not known to their registries export as other.
• A model exports by name only when it comes from the model catalog (built-in or models.dev). A model that exists only in your configuration exports as custom.
• Identity and enum values outside their owner's format are omitted.
• Any value containing a registered secret exports as invalid.
• Unregistered span names export as compozy.unknown.
• No exemplars are exported.

Never exported

The daemon never exports prompt or response text, tool arguments or results, file contents, filesystem paths, environment values, header values, credentials, error message text, span or attribute values outside the registries above, or incoming tracestate.

Forward to providers

Provider processes no longer inherit OTEL_* or TRACEPARENT from the daemon's process environment, even when daemon export is off. To keep independent provider telemetry configured before this change, move those variables into the agent's session_env. To share the daemon's collector settings instead, enable export and forward_to_providers as described below.

With forward_to_providers = true and export running, provider processes started afterwards receive:

OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_EXPORTER_OTLP_HEADERS=api-key=<resolved value>
OTEL_RESOURCE_ATTRIBUTES=compozy.session_id=sess-9c1e,compozy.agent=general
TRACEPARENT=00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01

TRACEPARENT is the provider's acp.process.spawn span, so a provider that honors it nests its own spans under the turn. It is set only when that span is sampled. Header values are resolved once at daemon start and percent-encoded.

This is an operator policy, not covered by the daemon privacy guarantee. The provider decides what it exports — some providers include prompt text or tool payloads in their own telemetry — and it receives the resolved header values. Whether a provider exports anything depends on that provider (some need their own enable flag, set through the agent's environment). Agent-defined OTEL_* values win. compozy status shows forwarding to providers while it is on.

Nothing is forwarded while export is disabled, inactive (observability.enabled = false), or degraded at start, for example by a header ref that does not resolve. Providers already running keep the environment they started with.

$ compozy config set observability.otel.forward_to_providers true
$ compozy daemon stop && compozy daemon start

Overhead

BenchmarkTurnTelemetryFootprint measures one turn with export disabled and with export enabled at a sample ratio of 1, exporting to an in-process OTLP/HTTP receiver with the daemon's database write instrumentation installed. Each turn runs through the ACP driver against a mock agent on a warm session: 200 streamed chunks, one hosted tool call, and a usage event. Reproduce it from the repository root:

Means of 10 runs of 10 turns each:

• Elapsed time is dominated by the mock agent's streaming, so it includes provider I/O.
• CPU is the benchmark process's user and system time from getrusage, around the measured turns and their final export. Individual runs ranged from 82 ms to 197 ms in both modes, so the CPU difference is within run-to-run noise. Allocations, which do not depend on machine load, grow by 563 per turn. The receiver runs in the same process, so the enabled CPU and allocation figures also include its decoding work; measuring with the receiver in a separate process is tracked in #1622.
• OTLP bytes are the serialized trace requests per turn plus one full metric push, counted as if every turn triggered a push.
• The spans are session.turn, acp session/prompt, and db.write spans for the turn's store writes. The benchmark runs the session runtime without the daemon's HTTP server and hosted tool service, so it does not count the request span, mcp.hosted.call, or tool.execute that a daemon turn also exports.

Measured on Linux amd64 with an Intel Core 7 240H (16 threads), with other agent sessions running (load average 3 to 7).

Troubleshoot export

Configuration errors stop compozy config set and config load with one of these messages:

Runtime failures never fail a command. They show as degraded in compozy status, with the error class per signal in last_error_class:

dropped_spans counts spans dropped because the export queue was full; a growing count means the collector cannot keep up.