OpenAI Evals Alternatives: Export by Oct 31, Gone Nov 30

OpenAI is shutting down its hosted Evals platform: existing evals become read-only on October 31, 2026, and "The Evals dashboard and API are scheduled to shut down" on November 30, 2026, according to the entry dated 2026-06-03 on OpenAI's deprecations page. Export your eval definitions, runs and per-row results through the API before October 31, then rebuild them in a tool you control. OpenAI's own migration guide recommends Promptfoo, and other OpenAI Evals alternatives such as Langfuse, DeepEval and Inspect AI can hold the same tests with some reshaping.

The eval runner isn't the only thing going: graders, Datasets, the prompt optimizer, the v1/prompts API and Agent Builder are also affected (details below). The dashboard does have an Export option, but according to a support reply in OpenAI's community forum it covers completed results, and only some eval types export as a Promptfoo config. OpenAI hasn't said how long run history is kept after shutdown, and doesn't mention the open-source openai/evals repository at all.

We build LLM features for clients in our Automation & AI Systems practice, and evals are part of every one of those builds. We don't sell or resell any eval tool. Every vendor fact below comes from the vendor's own pages, read on 2026-10-15, and the port in the middle of this post is our own run, with the configs and conversion script shown.

What exactly OpenAI is shutting down, and when

The hosted Evals product (dashboard and API) stops on November 30, 2026, and everything built on top of it goes with it. Here is the timeline as the deprecations page states it:

The other pieces are covered by banners on their own guides. The Evals guide and the Datasets guide both say: "Evals will become read-only for existing users on October 31, 2026, and the platform is scheduled to shut down on November 30, 2026." The prompt optimizer guide says "OpenAI is deprecating the dataset-backed prompt optimizer as part of the Evals platform", with the same two dates. That optimizer works from dataset annotations, output_feedback critiques and grader results, so it has nothing to run on once Datasets and graders are gone.

Graders are the fuzzy part. The deprecations entry says "Graders documented for eval workflows are part of this transition." The graders guide goes wider: "OpenAI is deprecating graders as part of the evals and fine-tuning workflows they support." It gives no dates. The deprecations entry adds that "Fine-tuning-related timelines remain covered in the self-serve fine-tuning section," and when we read that section on 2026-10-15, its dates (May 7, 2026, July 2, 2026 and January 6, 2027) didn't mention graders. If you use graders for reinforcement fine-tuning, get the date for your case from OpenAI in writing.

Two things OpenAI hasn't published, as of today:

• What "read-only" covers. OpenAI says existing evals content "remains available during the transition window", but it doesn't list which API calls stop working on October 31. Export everything before October 31 so your CI isn't depending on a read-only platform.
• How long eval runs, output items and report pages stay retrievable after November 30. Langfuse's migration guide reads the shutdown as meaning "the API is no longer available to read them"; that is Langfuse's reading, not OpenAI's. Treat it as unknown and assume they're gone.

Six months from announcement to shutdown is short for something wired into CI. For comparison, our API versioning post covers GitHub's published window of at least 24 months for an old REST API version.

Is the open-source openai/evals repo affected?

OpenAI hasn't said. The shutdown entry names "The Evals dashboard and API" and nothing else, and the deprecations page doesn't mention the GitHub repository.

github.com/openai/evals is a separate thing: the open-source framework OpenAI published in 2023, with its own registry of evals and model-graded templates, which you run locally. Checked with the GitHub API on 2026-10-15, it was not archived, its last commit was on 2026-04-14 ("Pin pre-commit hook revisions to immutable commits (#1644)"), and the commit before that was on 2026-04-06. GitHub reports its license as NOASSERTION, while the README says contributions fall "under the same MIT license as this repository."

The README carries no deprecation notice. It still says "You can now configure and run Evals directly in the OpenAI Dashboard," which stops being true on November 30, and "we are currently not accepting evals with custom code!"

So if your evals live in a local checkout of that repo, nothing in OpenAI's announcement switches them off. A framework with two commits so far in 2026 isn't somewhere to move new work. If you only used the hosted dashboard, the repo has no bearing on your migration.

How to export OpenAI evals before October 31

The dashboard has an Export option, but it won't give you everything. In an OpenAI community forum thread, a user found Export in an eval's "(…)" menu and got a JSONL download rather than a Promptfoo config. A reply from the OpenAI_Support account on July 3, 2026 explained that "A JSONL export is different. It only contains the results of a completed evaluation," and that "not every evaluation type currently supports exporting as a Promptfoo configuration." That's a support reply in the forum, not documentation, but it's the most specific thing OpenAI has said about export.

The cookbook Moving from OpenAI Evals to Promptfoo says "It does not require an OpenAI Evals export feature," and its path is to recreate your prompts, provider, tests and assertions by hand in promptfooconfig.yaml. That works if you have three evals and remember how they were built. If you have thirty, or you want your old scores as a baseline, the API is the complete path: definitions, datasets, runs and per-row output items, pulled while it still answers.

These are the endpoints that hold your data:

The output items are the part you can't recreate later. In the openai-python SDK (we checked the source at v3.31.0 on 2026-10-15), each output item has a datasource_item (the dataset row the run used), a sample (model input, output and token usage) and a results list in which each entry has a name, passed, score and type. That means the rows of a dataset can be rebuilt from its runs even if you've lost the original file.

This script writes all of it to disk. It uses the SDK's list methods, which page through results when you iterate over them. We haven't run it against a live Evals account (we have none to point it at), so test it on one eval first.

# export_openai_evals.py: dump eval definitions, runs and output items to ./evals-export
import json, pathlib
from openai import OpenAI

client = OpenAI()  # reads OPENAI_API_KEY
out = pathlib.Path("evals-export")

for ev in client.evals.list():
    d = out / ev.id
    d.mkdir(parents=True, exist_ok=True)
    full = client.evals.retrieve(ev.id)
    (d / "eval.json").write_text(full.model_dump_json(indent=2))
    for run in client.evals.runs.list(ev.id):
        (d / f"run_{run.id}.json").write_text(run.model_dump_json(indent=2))
        with open(d / f"run_{run.id}_items.jsonl", "w") as f:
            for item in client.evals.runs.output_items.list(run.id, eval_id=ev.id):
                f.write(item.model_dump_json() + "\n")

# dataset files uploaded for evals, including evals that never ran
files = out / "files"
files.mkdir(parents=True, exist_ok=True)
for fo in client.files.list(purpose="evals"):
    (files / f"{fo.id}.jsonl").write_bytes(client.files.content(fo.id).content)

The dataset format you're converting from is the one in the Evals guide: JSONL with one row per line, each wrapped in an item key, for example { "item": { "ticket_text": "My monitor won't turn on!", "correct_label": "Hardware" } }. The last loop in the script follows Langfuse's migration guide, whose export script "downloads every file uploaded with the evals purpose, so rows of evals that never ran are archived too." That is a vendor guide's claim and we haven't run it ourselves, so keep datasource_item from the output items as your fallback.

Keep the testing_criteria JSON next to the data. Every grader's type, operation, template strings such as {{ item.label }} and {{ sample.output_text }}, and pass_threshold live there, and those are what you rebuild next. The Datasets guide documents only exporting "your dataset to an Eval", not to a file, and the dashboard export described in the forum covers completed run results, so the API dump is the copy to keep.

How OpenAI graders map to Evals alternatives

Exact string graders map one to one almost everywhere; similarity and model-graded graders map to something with a different scale, which is where migrated evals silently change behavior. Neither OpenAI nor any of these vendors publishes a mapping, and the cookbook has no grader-type table at all; its advice for anything it doesn't cover is "Recreate it as a Promptfoo assertion and verify its behavior before relying on it." The table below is our mapping, built from each tool's own docs on 2026-10-15. Treat every row as a starting point you test.

Sources for the names: OpenAI's graders guide and API reference (label_model appears in the API reference, not the guide), Promptfoo's assertions reference, Langfuse's evaluator docs, DeepEval's custom metrics page and Inspect's scorers page. For the Langfuse threshold column, Langfuse's own migration guide says "Per-grader pass_threshold and passing_labels have no dedicated setting" and suggests keeping the raw score per item and computing the pass rate in a run-level evaluator.

A few rows deserve a warning before you trust them.

The string_check rows are safe. OpenAI's guide defines eq as "Returns 1 if the input matches the reference (case-sensitive), 0 otherwise", and Promptfoo's equals is "output matches exactly". Inspect's exact() is the exception: its docs say it will "Normalize the answer and target(s)" before comparing, so a case or whitespace difference that failed on OpenAI can pass there.

The fuzzy_match row is the one we'd expect to break, and it did, in the next section. OpenAI's guide describes it as "Fuzzy string match between input and reference, using rapidfuzz" and doesn't say which rapidfuzz scorer or how the score is scaled. Promptfoo's levenshtein is described as "Levenshtein distance is below a threshold." OpenAI's score is a similarity, where higher is better; Promptfoo's threshold is a distance, where lower is better.

Model-graded rows never port exactly. A score_model grader carries its own prompt, model and range; llm-rubric and G-Eval bring their own prompts, so the scores shift even when you paste the rubric across. Promptfoo's factuality and model-graded-closedqa are described as using the "Factuality method from OpenAI eval" and the "Closed QA method from OpenAI eval", which refer to the templates in the open-source repo, not the hosted score_model grader.

If your evals checked agent tool calls, Promptfoo's assertions reference lists is-valid-openai-tools-call and tool-call-f1, plus trajectory:tool-used, trajectory:tool-args-match and trajectory:tool-sequence for traced agent runs. We didn't test any of them.

Langfuse has a date of its own to watch. Its docs say that on Langfuse Cloud, trace-level evaluators stop producing results at the "v4 cutover on November 16, 2026"; self-hosted instances set their own timeline. Build migrated evaluators on observation-level evaluators.

Porting OpenAI Evals to Promptfoo: what broke

The exact-match grader ported cleanly in one pass; the fuzzy-match grader passed 3 of 8 rows after a literal port and 7 of 8 after we changed what the threshold meant. That second result is the reason to check every migrated threshold by hand.

On 2026-10-15 we took an 8-row ticket-triage eval in OpenAI's dataset format and moved it to Promptfoo 0.124.1 (Node v24.20.0, with PROMPTFOO_DISABLE_TELEMETRY=1). The model under test was a deterministic local stand-in: a keyword classifier and a fixed reply table, both plain JavaScript files. No OpenAI API was called. We wanted the migration mechanics, meaning row format, grader semantics and exit codes, and a real model would have added noise to exactly the thing being measured. The classifier has two deliberately naive rules ("can't" reads as a bug, "discount" isn't recognized as sales) so the eval had real failures to report.

The dataset, in OpenAI's {"item": {...}} shape (first two of eight rows):

{"item": {"ticket": "I was charged twice for my October invoice", "label": "billing", "reply": "Refund issued for the duplicate charge"}}
{"item": {"ticket": "Can't log in after resetting my password", "label": "account", "reply": "Password reset link sent again"}}

Promptfoo reads JSONL test files in which each row is {"vars": {...}}, and it has no documented OpenAI Evals importer. The conversion is a rename of the outer key, which took four lines:

// convert.mjs: OpenAI Evals rows ({"item": {...}}) to promptfoo rows ({"vars": {...}})
import fs from "node:fs";
const rows = fs.readFileSync(process.argv[2], "utf8").trim().split("\n").map((l) => JSON.parse(l));
fs.writeFileSync(process.argv[3], rows.map(({ item }) => JSON.stringify({ vars: item })).join("\n") + "\n");
console.log(`converted ${rows.length} rows`);

The template strings change too. OpenAI's {{ item.label }} becomes Promptfoo's {{label}}, because the row's fields are now top-level vars, and {{ sample.output_text }} disappears because Promptfoo assertions run against the output by default.

Run A: string_check eq ported to equals

The original grader was a string_check with operation eq, input {{ sample.output_text }} and reference {{ item.label }}. The port:

# promptfooconfig.yaml
description: Ticket triage, ported from OpenAI Evals
prompts:
  - "Classify this support ticket as billing, account, bug or sales: {{ticket}}"
providers:
  - file://classifier.mjs
defaultTest:
  assert:
    - type: equals
      value: "{{label}}"
tests: file://tests.jsonl

npx promptfoo eval --no-cache -o results-A.json reported 6 passed and 2 failed, and exited with code 100. The two failing rows:

FAIL: Can't log in after resetting my password | got "bug" | expected account | Expected output "bug" to equal "account"
FAIL: Do you offer a discount for nonprofits? | got "account" | expected sales | Expected output "account" to equal "sales"

Those are the two rules we planted, caught by the ported grader, which is what a one to one port should look like. Exit code 100 is the part that matters for CI. Promptfoo's command line docs say it "will return exit code 100 when there is at least 1 test case failure", and 1 for other errors; any CI step that treats a non-zero exit as failure will block the merge. The pass bar defaults to 100% and can be lowered with PROMPTFOO_PASS_RATE_THRESHOLD, and the code itself can be changed with PROMPTFOO_FAILED_TEST_EXIT_CODE.

Run B: text_similarity fuzzy_match, ported two ways

The second grader was a text_similarity with evaluation_metric fuzzy_match and pass_threshold 0.8, checking a drafted one-line reply against a reference reply. The stand-in replier returns small wording drift, the kind a fuzzy grader exists to tolerate. We put two assertions on every row: the literal copy of the old config, and the same intent written as a ratio.

# promptfooconfig-similarity.yaml (assertions only)
defaultTest:
  assert:
    # 1. Literal copy: levenshtein's threshold is a maximum edit distance
    - type: levenshtein
      value: "{{reply}}"
      threshold: 0.8
      metric: literal_copy
    # 2. Same intent as a ratio: 1 - distance / longer length >= 0.8
    - type: javascript
      value: |
        const a = output, b = context.vars.reply;
        const d = Array.from({ length: a.length + 1 }, (_, i) => [i]);
        for (let j = 1; j <= b.length; j++) d[0][j] = j;
        for (let i = 1; i <= a.length; i++)
          for (let j = 1; j <= b.length; j++)
            d[i][j] = Math.min(d[i-1][j] + 1, d[i][j-1] + 1, d[i-1][j-1] + (a[i-1] === b[j-1] ? 0 : 1));
        const ratio = 1 - d[a.length][b.length] / Math.max(a.length, b.length);
        return { pass: ratio >= 0.8, score: ratio, reason: `ratio ${ratio.toFixed(3)}` };
      metric: ratio_0_8

npx promptfoo eval -c promptfooconfig-similarity.yaml --no-cache -o results-B.json exited 100 again. Per row:

Promptfoo's reason on the first row says it plainly: "Levenshtein distance 4 is greater than threshold 0.8". A distance threshold of 0.8 only lets through a distance of 0, so the literal copy turned a fuzzy grader into an exact-match grader without any warning. Nothing errored and the config validated. The only symptom was a pass rate that dropped from what you were used to, which is easy to read as a model regression.

Rewriting the threshold as a ratio brought back the tolerance the original grader was meant to have: 7 of 8. The row that failed both ways reorders words ("Sent nonprofit discount details"), and character edit distance punishes reordering hard, at 11 edits. Whether OpenAI's grader passed that row is something we can't tell you. Note also the third row, at 0.806: one more changed character and it fails. Small formula differences matter right at the threshold, which is where your borderline rows sit.

OpenAI's guide says fuzzy_match uses rapidfuzz but not which scorer or scaling, and with no Evals account to run it on, we couldn't confirm the formula. Our ratio is an approximation. The fix that holds up is calibration: export a sample of past fuzzy_match scores from your output items before shutdown, compute your new metric on the same rows, and pick the threshold that reproduces the old pass and fail decisions. If you can match rapidfuzz itself, a python assertion that calls it is closer still, but check the scorer against your exported scores either way.

Where to migrate from OpenAI Evals

Pick by where you want evals to run and who needs to see the results; on shape alone, any of these can hold a migrated eval.

The table is alphabetical and covers the facts each vendor states on its own pages. It isn't a ranking.

One disclosure about the tool OpenAI recommends. Promptfoo's blog post Promptfoo is joining OpenAI, dated March 9, 2026, announced the deal, and the Promptfoo README now says "Promptfoo is now part of OpenAI." So OpenAI's recommended replacement is an OpenAI-owned tool. The same README says "Promptfoo remains open source and MIT licensed" and "LLM evals run 100% locally - your prompts never leave your machine". Telemetry is on by default; its telemetry page says it "does not include prompts, model outputs, test cases, provider API keys, or full configuration files", and PROMPTFOO_DISABLE_TELEMETRY=1 turns it off.

Some situations, and the tools whose documented features fit them.

If your evals are a CI gate and you want them in the repo next to the code, Promptfoo, DeepEval and Inspect AI all run as local commands. Promptfoo is the one OpenAI documents, and our port above needed only a key rename to read OpenAI's rows; we didn't test the same conversion for the other tools. DeepEval fits a team that already writes pytest. Inspect AI, from the UK AI Security Institute and Meridian Labs, is built for research-style evaluation with sandboxes, multiple scorers and reducers, and reads JSON Lines datasets directly.

If product managers or reviewers used the OpenAI dashboard to look at failures, you need a hosted UI again. Langfuse can be self-hosted for free or used as a cloud service, and also traces production calls. Braintrust and LangSmith are hosted platforms whose self-hosting is limited to their enterprise plans. Arize Phoenix is free to self-host, but its Elastic License bars offering it "as a hosted or managed service", which matters only if you resell it.

If your data can't leave your infrastructure, the local tools and the self-hosted Langfuse or Phoenix keep it there; with Braintrust's enterprise option, Braintrust says its hosted control plane "does not store or process your sensitive data".

A common setup has two layers: a local runner that gates merges and a dashboard for reviewing outputs. It matches the split we describe for assistants in how to build an AI assistant, where a 50-question eval set from real traffic comes before any launch, and for agent features in adding AI agents to a SaaS product, where every human correction during assist mode feeds the eval suite.

A migration checklist for the next six weeks

Export first, run in parallel while you can, and switch CI over well before the API goes dark.

Before October 31 (read-only begins):

1. List every eval and every CI job, script or scheduled task that calls the Evals API. Search the codebase for evals.runs.create and /v1/evals.
2. Run the export script above. Keep eval.json, the run files and the output items for every eval you still care about, including old runs you'd use as a baseline.
3. Freeze the evals. No new graders or edits on OpenAI from here; new work goes into the new tool.
4. Convert the datasets (item to your new tool's row shape) and rebuild the exact-match graders, which should port one to one.

By November 30 (shutdown):

1. Rebuild similarity and model-graded graders, then calibrate each threshold against the exported scores until the pass and fail decisions on old rows match.
2. Run old and new side by side on the same model for at least one release while the old runs are still readable, and investigate every row where they disagree.
3. Switch the CI gate to the new runner, and check that a failing test actually fails the build (exit code 100 in Promptfoo's case).
4. Move anything on v1/prompts reusable prompts into application code, as OpenAI's page says, and plan an Agent Builder replacement if you use it.

After November 30:

1. Delete the dead API calls and the secrets that only existed for them.
2. Keep the export archive. Langfuse's migration guide says "Historical runs are not imported as experiments", and a hand port to Promptfoo doesn't bring them over either. If someone asks in March why a score moved, it's the only record of the old grader's behavior.
3. Write down who owns the eval suite now. Our AI development cost guide makes the same point for agency builds: own the code, the prompts and the evaluation suite at handoff.

What this post doesn't cover

We didn't run any OpenAI grader, so every equivalence in the mapping table is inferred from documentation, and our ratio formula approximates fuzzy_match and may score borderline rows differently. We ported one eval to one tool against a stand-in model; we didn't port to Langfuse, Braintrust, LangSmith, Phoenix, DeepEval or Inspect AI, and we didn't compare their pricing or how their model-graded scores differ. Reinforcement fine-tuning graders follow dates we couldn't confirm. Vendor pages change, and this post reflects what they said on 2026-10-15.

If you'd rather hand the port to someone who has done it, our Automation & AI Systems team rebuilds eval suites as part of the LLM features we build and run for clients.