Find errors in production agent traces
Hi HN, we’re Robert and Din from Laminar (https://laminar.sh). Task: Hi HN, we’re Robert and Din from Laminar (https://laminar.sh). We’ve been building open-source observability for AI agents since 2024. Recently, we released flow-1, a model trained with RL to find errors in agent traces. On our benchmark (https://laminar.sh/blog/flow-1), it matches GPT-6-sol in detection quality while being 23x cheaper. This makes it affordable to analyze every production agent trace, without sampling.Here’s a one-minute walkthrough: https://youtu.be/KJa3h6ZJNLQWe also put together a post with example traces and analyses from flow-1 and other models: https://laminar.sh/blog/flow-1-in-actionAgent observability is crowded, so instead of a bullet list of how Laminar differs from X, we’ll explain how we’re solving agent observability from first principles.Agents now run for many steps and fail implicitly. An agent can click the wrong button or cite the wrong document while every call completes successfully. These errors are the most useful ones to catch and understand. To find them, you need to understand what the agent was trying to do and inspect its LLM calls and tool results. Simply looking at the final agent output is not enough.Collecting traces alone doesn’t solve this. A lot of agent observability platforms are still data collectors at their core. You can end up paying to store traces that nobody has time to read or extract value from.Using another LLM to inspect traces is THE solution, but capable models are expensive at scale and cheaper ones often miss errors or flag unrelated things.So we built our model to solve exactly this problem. We finetuned a small OSS model and then post trained it with RL on thousands of synthetic investigations of realistic agent workflows in software, legal, customer support, etc. We got a model that is both good at agent trace analysis and very cheap to run.flow-1 runs inside a harness we call Signals. It acts as a coding agent, treating an agent trace like a repo, with each span (an LLM call, tool call, etc.) as a file. It starts from a compressed string representation of a full agent trace and a user prompt, then greps across spans and reads the relevant ones in full.The beauty of Signals is that it can respond to a general prompt. You can ask it to identify user frustration, check whether your browser agent completed a checkout, and so on. It’s not limited to raw error detection. It returns findings as JSON in a schema users can freely define. We call these signal events.Once you’re analyzing every trace, you also have too many individual events to read. That’s why we automatically group similar signal events into clusters. You can see recurring problems and how often they happen, then drill into individual events and the traces behind them.You can use traces from a cluster to build an eval dataset for that type of failure. You can then use your favourite coding agent to hillclimb on those evals and check that the fix works across these cases before shipping it to prod.Trace → analyze → catch errors and form clusters → fix → release. Making this loop easy is why we built Laminar.We’ve put the same thought into the rest of Laminar. Our blog (https://laminar.sh/blog) explains how looking closely at how agents actually work led us to build 20x trace compression, a custom SQL query engine and automatic task extraction.Laminar is Apache 2.0 and self-hostable (https://github.com/lmnr-ai/lmnr). We make money from the managed cloud.We’d love to hear how you track your agents’ performance today. How much do you learn from user feedback versus your own monitoring? And when you fix something, how do you check that it hasn’t introduced a different failure?; tools: custom-built or unnamed agent. Human role: The original post does not say how a person stayed involved.; limitations: Prerequisites include self-hosting Laminar (Apache 2.0) or using their managed cloud. The model's performance relies on synthetic RL training data, which may not generalize to all agent workflows. Cost savings are vendor-reported; actual infrastructure costs for trace storage and processing are not detailed. Human validation is required to filter false positives before building eval datasets..
Tools
custom-built or unnamed agent
Published on Agent Work Atlas:
Overview
Hi HN, we’re Robert and Din from Laminar (https://laminar.sh).
Workflow
Hi HN, we’re Robert and Din from Laminar (https://laminar.sh). We’ve been building open-source observability for AI agents since 2024. Recently, we released flow-1, a model trained with RL to find errors in agent traces. On our benchmark (https://laminar.sh/blog/flow-1), it matches GPT-6-sol in detection quality while being 23x cheaper. This makes it affordable to analyze every production agent trace, without sampling.Here’s a one-minute walkthrough: https://youtu.be/KJa3h6ZJNLQWe also put together a post with example traces and analyses from flow-1 and other models: https://laminar.sh/blog/flow-1-in-actionAgent observability is crowded, so instead of a bullet list of how Laminar differs from X, we’ll explain how we’re solving agent observability from first principles.Agents now run for many steps and fail implicitly. An agent can click the wrong button or cite the wrong document while every call completes successfully. These errors are the most useful ones to catch and understand. To find them, you need to understand what the agent was trying to do and inspect its LLM calls and tool results. Simply looking at the final agent output is not enough.Collecting traces alone doesn’t solve this. A lot of agent observability platforms are still data collectors at their core. You can end up paying to store traces that nobody has time to read or extract value from.Using another LLM to inspect traces is THE solution, but capable models are expensive at scale and cheaper ones often miss errors or flag unrelated things.So we built our model to solve exactly this problem. We finetuned a small OSS model and then post trained it with RL on thousands of synthetic investigations of realistic agent workflows in software, legal, customer support, etc. We got a model that is both good at agent trace analysis and very cheap to run.flow-1 runs inside a harness we call Signals. It acts as a coding agent, treating an agent trace like a repo, with each span (an LLM call, tool call, etc.) as a file. It starts from a compressed string representation of a full agent trace and a user prompt, then greps across spans and reads the relevant ones in full.The beauty of Signals is that it can respond to a general prompt. You can ask it to identify user frustration, check whether your browser agent completed a checkout, and so on. It’s not limited to raw error detection. It returns findings as JSON in a schema users can freely define. We call these signal events.Once you’re analyzing every trace, you also have too many individual events to read. That’s why we automatically group similar signal events into clusters. You can see recurring problems and how often they happen, then drill into individual events and the traces behind them.You can use traces from a cluster to build an eval dataset for that type of failure. You can then use your favourite coding agent to hillclimb on those evals and check that the fix works across these cases before shipping it to prod.Trace → analyze → catch errors and form clusters → fix → release. Making this loop easy is why we built Laminar.We’ve put the same thought into the rest of Laminar. Our blog (https://laminar.sh/blog) explains how looking closely at how agents actually work led us to build 20x trace compression, a custom SQL query engine and automatic task extraction.Laminar is Apache 2.0 and self-hostable (https://github.com/lmnr-ai/lmnr). We make money from the managed cloud.We’d love to hear how you track your agents’ performance today. How much do you learn from user feedback versus your own monitoring? And when you fix something, how do you check that it hasn’t introduced a different failure?
Reported result
The source claims flow-1 matches GPT-6-sol in detection quality while being 23x cheaper, enabling full-trace analysis without sampling. However, no independent benchmarks or measured business outcomes are provided. The vendor reports that the model can distinguish between environment failures and reasoning errors, but this capability remains unverified by external data.
Human role
The original post does not say how a person stayed involved.
Limitations
Prerequisites include self-hosting Laminar (Apache 2.0) or using their managed cloud. The model's performance relies on synthetic RL training data, which may not generalize to all agent workflows. Cost savings are vendor-reported; actual infrastructure costs for trace storage and processing are not detailed. Human validation is required to filter false positives before building eval datasets.
Try it
Act as an AI observability engineer. I need to set up a trace analysis workflow using Laminar's flow-1 model. 1. **Context**: We are running [replaceable: e.g., customer support agents] and need to detect implicit failures like wrong tool usage or hallucinated citations. 2. **Setup**: Configure the Signals harness to ingest traces from [replaceable: source system/URL]. Define a JSON schema for signal events that captures [replaceable: specific error types, e.g., 'checkout_failure', 'wrong_document_cited']. 3. **Execution**: Run flow-1 on a sample of [replaceable: number] recent production traces. Use the prompt: 'Identify any spans where the agent failed to achieve the user's goal despite successful API calls.' 4. **Output**: Generate a report listing detected signal events, grouped by cluster. For each cluster, provide 3 example traces and the model's reasoning. 5. **Validation**: Before applying this to all traces, I will manually review the sample report. Do not proceed to full-scale analysis or create eval datasets until I confirm the false positive rate is acceptable. 6. **Error Handling**: If the model flags environment errors (e.g., 403s) as reasoning failures, adjust the prompt to explicitly ignore infrastructure issues. Please start by generating the configuration code for the Signals harness and the initial sample analysis script.
Source reports are not independently verified. Publication date is the catalog publication date, not the original article date.