How to Monitor LLMs in Production

You shipped your AI feature. It works in staging. Users are hitting it. Now what?

Most teams add console.log and call it monitoring. That works until your LLM costs spike 300% overnight, response quality silently degrades, and users leave without telling you why.

Real LLM monitoring has five levels. Most teams never get past level two.

Level 1: Basic Logging

What: Log every LLM call: input, output, model, and timestamp.

You get: A record of what happened. If something breaks, you can search logs and find it.

You miss: Everything quantitative. You know GPT-4 was called 10,000 times yesterday, but not how much that cost, how long it took, or whether users found the responses useful.

Tools at this level: Application logs, CloudWatch, Datadog basic.

Almost every team starts here. The problem is that many stay here.

Level 2: Metrics

What: Track quantitative measurements: token counts, latency (P50/P95/P99), error rates, cost per call, and model distribution.

You get: Dashboards. You can see trends, set alerts, and identify anomalies. "GPT-4 P95 latency increased 40% this week."

You miss: Context. You know latency spiked, but not why. You know costs are up, but not which workflow step or which user cohort is driving it.

Tools at this level: Prometheus + Grafana, Langfuse basic, Helicone.

This is where most "AI observability" tools operate. It's necessary but insufficient.

Level 3: Distributed Traces

What: Trace the full lifecycle of a request across multiple services and model calls. Each trace shows the complete journey: API request → preprocessing → model call → postprocessing → response.

You get: Cause and effect. "This request was slow because the RAG retrieval step took 3 seconds, not the LLM call." You can see where time is spent, which model was chosen, and how each step contributed to the final output.

You miss: The user perspective. Traces show what your system did. They don't show what your user experienced or whether the response was useful.

Tools at this level: OpenTelemetry, Jaeger, Arize Phoenix, Langfuse traces.

Level 3 is where engineering teams start to feel confident. But there's a gap between system performance and user satisfaction that traces don't bridge.

Level 4: Workflow Control Context

What: Connect LLM performance data with workflow context. Which endpoint called the model? Which route was selected? Did the run retry? Did it hit an approval gate? Which version was deployed?

You get: The missing link. "The premium route is triggered by low-confidence extraction runs." "The retry spike comes from version 12 of the support workflow." "The default path is cheap, but the escalation path is consuming 42% of spend."

You miss: The ability to change behavior without shipping more glue code. You have the data. You need routing, approval, and version controls.

Tools at this level: Custom control-plane code around Langfuse, Helicone, OpenTelemetry, queues, API gateways, and provider SDKs.

This is where operational clarity jumps dramatically. But almost nobody gets here cleanly because it requires joining AI metrics, workflow state, API deployment, routing policy, and cost data.

Level 5: Controlled Routing and Recovery

What: Use monitoring data to control execution. Route simple work to efficient models, escalate risky work to premium reasoning or human review, retry transient failures, and version each workflow safely.

You get: Actionable execution control. "Route confidence > 0.8 to the efficient path." "Escalate policy-flagged runs to review." "Retry provider timeouts twice, then fail over to the local model." "Keep version 12 behind 10% of traffic until error rate stabilizes."

What makes this different: Levels 1-4 give you data and dashboards. Level 5 changes how the workflow behaves in production.

Tools at this level: Almost nothing on the market does this comprehensively.

The Monitoring Stack Problem

Here's what a Level 4-5 monitoring stack looks like if you build it yourself:

| Layer | Tool | Cost | | -------------------- | ---------------------- | ---------------------------- | | Logging | ELK Stack / Datadog | $$-$$$$ | | Metrics | Prometheus + Grafana | $ (on-premise) | | Traces | OpenTelemetry + Jaeger | $ (on-premise) | | LLM-specific metrics | Langfuse / Helicone | $$-$$$ | | Workflow state | Custom database | Engineering time | | Routing layer | Custom code | Engineering time | | Approval/retry logic | Custom services | Significant engineering time |

That's 5-7 tools, 3-4 data pipelines, and weeks of integration work. The control layer between AI metrics and production behavior is the custom code nobody has time to build.

The Unified Approach

The reason most teams stay at Level 2 isn't laziness; it's fragmentation. Each level requires different tools, different data formats, and different expertise. Unifying them is an engineering project in itself.

Sinapsis AI collapses all five levels into one control plane:

• Levels 1-2: Built-in logging and metrics for every model call, with per-step cost tracking
• Level 3: OpenTelemetry-native distributed tracing across workflows
• Level 4: Workflow context: endpoint, version, route, retry, approval, and cost data tied to each run
• Level 5: Controlled routing and recovery with efficient paths, premium paths, retries, and escalation gates

No tool sprawl. No custom correlation pipelines. No weeks of integration.

Getting Started

If you're at Level 1-2, here's the practical path forward:

1. Add per-step cost tracking. This alone changes how teams think about AI spending.
2. Implement distributed traces. OpenTelemetry is the standard. Instrument your AI calls.
3. Connect workflow context. Track endpoint, version, route, approval state, retry state, and output status.
4. Control execution. Use the data to route, retry, escalate, or roll back safely.

Or use a platform that gives you all five levels from day one.

Monitor what your models do. Monitor how workflows execute. Control the route between the two.