Rokan and Braintrust
Braintrust logs what your agent does in production, turns those traces into evals, and has an assistant, Loop, that suggests prompt improvements. Rokan works per customer: why the agent failed at that customer, the fix (including code fixes as a branch), and a replay of the fix on every customer's past runs before it ships. Rokan reads Braintrust logs as JSON, so you keep Braintrust and add Rokan next to it.
What Braintrust does
Braintrust's pitch is to "Discover patterns in production, turn them into evals, and improve quality with every release." You inspect every trace and tool call, score outputs "with LLMs, code, or humans", and turn production traces into versioned datasets. An automation called Patterns finds recurring issues across production traces. Loop, which Braintrust calls "AI that helps you improve agents", can write scorers, build datasets, explain poor scores from your logs and suggest prompt improvements (Braintrust docs).
If you want a strong eval workflow, with production data feeding your tests, Braintrust covers that.
What Rokan adds
- The why, for one customer. Rokan reads one customer's failing runs with that customer's own rules, hours and config, and says why the agent failed there and not elsewhere.
- The fix, written. A prompt or config edit, or a code change as a branch in your repo.
- A replay on every customer. Before the fix ships, Rokan replays it on every customer's past runs. If it would break a run that works today at another customer, it is blocked or scoped to the one customer.
- Done for you. You send runs and get back a report and a fix. A person on your team approves every fix.
When Braintrust alone is enough
- Your failures are mostly prompt problems, and Loop's suggestions plus an experiment on your datasets get you to a fix.
- Your eval datasets already include runs from each customer, so a regression at one customer shows up before release.
- Your team has time to own the eval suite and keep it current.
When you'd add Rokan
- The agent fails at one customer and works at the others, and the cause is in that customer's setup.
- The fix needs a code change, not only a prompt edit.
- A fix for one customer keeps breaking another, and nobody has time to build and maintain per-customer test sets.
How to send Braintrust logs to Rokan
The Braintrust API returns project logs as JSON from GET /v1/project_logs/{project_id}/fetch, with a limit on how many traces come back. Each event includes its input, output, scores and metadata. For more complex filters, Braintrust points to its BTQL endpoint (Braintrust docs). Save the result as a JSON file.
Send 20 to 50 runs, from more than one customer if you can, plus your prompt and config. If the metadata doesn't show which customer a run belongs to, tell us the field in one line. The steps are on the trial page.
What Rokan does not do
Rokan's replay is a model playing the user's side against your prompt and rules. It does not call your tools or your voice stack. Rokan is also early: it has no paying customers yet.