Building an eval platform

Evals seem to be easy at the start. You give an agent input, it returns output, and you run a script to see if it meets your expectations. This is true for research projects, not an actual production setting. In real work, you have your system where the agents need to run, your teammates, deadlines, and resource constraints, which introduce a number of issues:

1. Inconsistency - every dev writes evals their own way, so it's hard to pick up where someone else left off.
2. Data drift - an eval works only on your laptop and only until you refresh the DB.
3. Environment setup - you have a DB, MCP server, object storage, agent sandboxes, etc.
4. Scaling - you can run only so many evals in parallel before your laptop runs out of resources.
5. Scoring - now everyone writes evals, but not everyone has studied ML, so you end up with the wrong metrics.

At Hypha, our agents read documents, query databases and call tools through our app. Evaluating them means reproducing the data, files and application around the model. Our stack probably looks familiar to you:

• Supabase — database and document storage
• Next.js on Vercel — application and MCP server
• E2B — agent sandboxes
• Langfuse — traces and eval datasets
• OpenAI / Anthropic / Vercel AI Gateway — models and model routing

Starting with an interface

We initially just added the Langfuse SDK to the codebase and declared the eval platform ready. After a few weeks, we had a dozen siloed evals, all running their own way and measuring their own metrics. To tackle this problem, I defined an interface:

The shared interface for defining an eval.

Env points to the frozen branch in Supabase, which solves data drift. More on this in the next paragraph.

const env = {
  branch: 'snap-document-classification-v1',
  orgs: [{ orgId: 'org-1', orgName: 'Example Co' }],
}

Dataset defines eval cases through a function. This allows us to use dynamic logic for retrieving data from an external source, preprocessing it, etc. Each case contains the input given to the experiment and the expected output used for scoring.

const dataset = {
  name: 'document-classification',
  async getItems() {
    return [{
      input: { documentId: 'doc-1' },
      expectedOutput: { documentTypes: ['Rent Roll'] },
    }]
  },
}

Experiment is a function that runs the experiment we are evaluating.

const experiment = async ({ input }) => {
  const result = await classifyDocument(input)
  return {
    text: JSON.stringify({ documentTypes: result.documentTypes }),
  }
}

Evaluator computes scores per dataset item, like accuracy against a rubric for the generated text. And RunEvaluator computes metrics across the full run, such as micro F1.

The Eval definition connects the environment, dataset, experiment, and evaluators:

export default {
  name: 'document-classification',
  env,
  dataset,
  experiment,
  evaluators: [new JsonDiffEvaluator()],
}

Solving data drift

Evals need to work not only when you develop something for the first time, but also months down the road, so you can evaluate switching to newer models and catch regressions when you modify your agent. But your data model isn't static. A seed script you wrote for a DB today will have some constraint issues in a week. Fixing the seed script every time you try to run an eval is too much effort. You need to keep it up to date as migrations come in. I solved it with Supabase branches:

Supabase branches for the application and eval snapshots.

We have 2 Supabase projects:

1. Application - has prod, stg, PR preview and eval run branches.
2. Eval snapshots - has a branch per eval environment.

The lifecycle looks like this:

1. Snapshot saved - during eval creation, a developer freezes their local Supabase instance as an environment snapshot, including both structured data and files. The snapshot is stored in a separate branch in the eval snapshots project. We limited snapshot size to 1 GB to keep eval setup and iteration fast.
2. Migration reaches staging - a PR introduces a schema change. This seed script would break.
3. Snapshots migrated - the same migration is applied to every eval snapshot, keeping their schemas in sync with staging.
4. Snapshot copied into a run - when the eval runs, the snapshot is copied into a local database or any other Supabase branch.

Adding dev tooling

To prevent developers from writing ad hoc scripts for their evals, we need dev tooling. Just a simple CLI tool to craft their environment, upload the dataset to Langfuse and execute evals.

Environment

• evals org pull --org-id <id> --source-url <url> - copy organization data locally to prepare the cases.
• evals env create <eval> --source-url <url> - freeze the prepared environment into a snapshot.
• evals env load <eval> - restore its frozen database and document files locally.

Dataset

• evals sync <eval> - preview changes to the dataset in Langfuse.
• evals sync <eval> --apply - apply those dataset changes.

Running evals

• evals list - find an eval.
• evals run <eval> --experiment <name> - run it against the loaded environment.
• evals run <eval> --keys <id> - run a single case (for debugging).
• evals run <eval> --pr <number> — run an eval in the cloud against a PR (see the next section).

Moving evals to the cloud

I've had my 48 GB M4 Pro MacBook black out so many times while running parallel evals. And it's impossible to do any research at scale if you can only run 2–3 evals in parallel. A simple context optimization for OpenAI's Decision API took over 60 eval runs. Running it locally would take days and make my laptop unusable. So, using all the existing primitives, we built cloud evals:

Cloud eval setup and the PR environment it runs against.

GitHub Actions creates a Supabase branch and loads the eval’s snapshot into it. It then deploys the PR’s application to Vercel, pointing it at that branch. The eval runs on GitHub Actions, with agents running in E2B and results recorded in Langfuse.

Creating a framework

Time to address the scoring issues. Development now mostly centers around agents, and SWEs who have never studied ML are creating evals. Sometimes this leads to the wrong metrics being used. A common example would be using accuracy instead of F1.

Suppose a document contains values for only 2 of 20 fields. The agent leaves every field empty. Its accuracy is 90%, because 18 fields are correctly empty. Its F1 is 0%, because it extracted neither of the actual values.

For exactly this reason, we added Evaluator and RunEvaluator to the abstraction to arm all developers on the team with a set of well-designed and tested evaluators. We currently have:

• LLM Judge - accepts text and a list of yes/no questions with points. It answers the questions against the text and returns a score based on the number of points earned out of all possible points. I tested and tuned the judge’s prompt against known correct and incorrect answers, so the team could reuse a tested judge.
• JsonDiff - accepts expected and actual JSON, compares the values and returns a similarity score. It can also compute micro, macro, weighted or sample F1 across the run.

const jsonDiff = new JsonDiffEvaluator({ f1: ['micro', 'samples'] })

const eval: Eval = {
  name: 'example',
  env,
  dataset: dataset,
  experiment: experiment,
  evaluators: [jsonDiff, new BinaryRubricLlmJudge()],
  runEvaluators: jsonDiff.runEvaluators, // micro and samples F1
}

---

Introducing a proper eval framework took a few weeks and a lot of effort from me and @eshwar. But it unlocked so many research projects: a 300-run ablation study on how to build efficient agent tools, experiments with Decision Models to replace an agent for document classification, the development of new Hypha products and regular model upgrades. The only issue left to solve is data quality in evals, particularly in expected outputs. Up to this point, all the data curation was done by developers and coding agents, both of whom have limited knowledge of private finance. To solve this, I built an annotation engine allowing our GTM team to review and label data.