Concept 2 min

Monitoring (Traces)

Eval happens before, in the lab. Monitoring happens after, in production. You need both.

In one sentence

The record of what the agent did at each stage (prompts, tool calls, cost, latency, failures) while it runs in production.

Before

Agent running in production

What it does

Logs prompt, tool, cost, and latency

After

Anomaly visible before the sponsor sees it

↺what appears here becomes a case in the answer key

Analogy

The airplane’s black box. It doesn’t prevent the accident, but it makes it possible to understand what happened and keep it from happening again.

Example

The dashboard shows, per letter: cost (R$ 0.42), time (38s), how many times the agent rewrote during self-check, and whether it escalated to a human.

In the third week, the average cost jumped to R$ 1.10. The trace showed the cause in two minutes: field reports scanned as images were being loaded in full into the context. Without trace, the line would have been "the bill went up and we don’t know why", and the likely response would have been to shut down the project.

The common mistake

Confusing eval with monitoring. Eval happens before, in the lab, with known cases and questions you choose. Monitoring happens after, in production, with real cases that bring their own questions.

If you only have eval, you discover the problem through the sponsor. If you only have monitoring, you discover it after it has already been sent. Both are necessary, and what monitoring catches in production should become a case in the golden dataset.

In practice

Instrument in the first week. If the trace only starts after the first incident, it won’t help you understand that incident.
Log per run: input, output, tools called, tokens, cost, latency, and prompt version.
Langfuse, LangSmith, or a table in your database. The format matters less than having the record.
Three simple alerts catch almost everything: average cost 2× above the previous week, escalation rate rising, p95 latency blowing past the limit.

How to make it tangible

Log per run (input, output, tools, cost, latency) plus three simple alerts.

Connects with

Translated from Portuguese with AI assistance.