Rokan and LangSmith
LangSmith traces your agent step by step, monitors it in production, runs evals and groups traces into failure modes. Rokan takes a failure at one customer and does the next part: why it failed there, the fix, and a replay of the fix on every customer's past runs before it ships. Rokan reads LangSmith runs, so you keep LangSmith and add Rokan next to it.
What LangSmith does
LangSmith, from LangChain, promises to help you "Know what your agents are really doing". It traces each step so you can find what hurts latency, cost and response quality, gives a real-time view of agents in production, and has an Insights feature that groups traces to "detect usage patterns, common agent behaviors, and failure modes". It also covers evaluation and deployment, and it says it "works with any LLM framework", not only LangChain.
If you want to see what your agent did, where it fails most and how it scores, LangSmith covers that.
What Rokan adds
- The why, for one customer. A failure mode tells you what went wrong across many runs. Rokan reads one customer's failing runs with that customer's own rules, hours and config, and says why it failed there.
- The fix, written. A prompt or config edit, or a code change as a branch.
- A replay on every customer. Before the fix ships, Rokan replays it on every customer's past runs. If it would break a run that works today at another customer, it is blocked or scoped to the one customer.
- Your approval. A person on your team approves every fix.
When LangSmith alone is enough
- Your customers use the agent in much the same way, so a fix that works for one works for all.
- Insights points at the problem and your team writes the fix the same day.
- You run evals on a dataset from each customer before every prompt change. That covers much of what Rokan's replay does.
When you'd add Rokan
- The agent fails at one customer and works at the others.
- Each customer has its own rules or setup, and a fix for one keeps breaking another.
- You can see the failure in LangSmith but nobody has the hours to work out the cause and test a fix across every customer.
How to send LangSmith runs to Rokan
The LangSmith SDK fetches runs with client.runs.query, which replaces list_runs. It takes project IDs, and it only looks back one day unless you pass min_start_time (LangSmith docs). Save the runs as JSON. For large volumes, LangSmith also has a bulk export to an S3-compatible bucket, but a trial needs far less than that.
Send 20 to 50 runs, from more than one customer if you can, plus your prompt and config. If the runs don't show which customer they belong to, tell us the field in one line. Not sure how to export? Paste three runs that went wrong into the email. The steps are on the trial page.
What Rokan does not do
Rokan's replay is a model playing the user's side against your prompt and rules. It does not call your tools or your voice stack. Rokan is also early: it has no paying customers yet.