Using a coding agent? Point it at /agents.md. It has the exact setup steps and says what needs your approval. Also available: /llms.txt and the OpenAPI spec.

Connect your app

Wrap your Anthropic or OpenAI client where you create it. The SDK never throws, never blocks your call and does not change what the client returns. Without a key it sends nothing.

npm install @swapvera/sdk

import Anthropic from '@anthropic-ai/sdk'
import { monitor } from '@swapvera/sdk'

const anthropic = monitor(new Anthropic())   // reads SWAPVERA_KEY

Python, included in the swapvera package:

pip install swapvera

from anthropic import Anthropic
from swapvera.sdk import monitor

client = monitor(Anthropic())   # reads SWAPVERA_KEY

It captures create and stream calls, with or without stream: true, including client.beta.messages. A streamed call is sent only when your app reads it to the end. OpenAI clients work the same way, and so do the async Python clients. Options: key, endpoint, sample (share of calls to capture, 0 to 1, default 1), feature and debug. In JavaScript also disabled, timeoutMs and waitUntil; on Vercel, pass waitUntil from @vercel/functions so uploads finish after the response.

Calls are grouped by system prompt. If one prompt serves several tasks, name them so they are compared apart:

const anthropic = monitor(new Anthropic(), {
  feature: (params) => params.metadata?.task   // any string you can derive from the call
})

Not captured: calls with images or files, which cannot be replayed from text. Gemini clients are not wrapped yet: use the CLI, which also fits if your data must stay on your machines.

Check that calls arrive: open the dashboard, or:

curl -H "Authorization: Bearer $SWAPVERA_KEY" https://swapvera.com/v1/project

A feature can be compared from 10 captured calls. Around 150 distinct requests give a verdict room to reach “safe”.

CLI quickstart

pip install swapvera
swapvera init my-check && cd my-check
export ANTHROPIC_API_KEY=...        # or OPENAI_API_KEY / GOOGLE_API_KEY
export SWAPVERA_LICENSE=sv_...    # from your email
swapvera run --out report.md

Edit swapvera.yaml to set the model you use today (from) and its replacement (to, or auto for the official one), and put your cases in cases.jsonl.

Scanner

Free and open source. Finds model IDs with an announced shutdown in your code, configuration and .env files.

# .github/workflows/swapvera.yml
name: Swapvera
on:
  push: { branches: [main] }
  schedule: [{ cron: '0 7 * * 1' }]
permissions: { contents: read, issues: write }
jobs:
  scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: pixelric-spec/swapvera-scanner@v1
        with:
          fail-on: retired   # retired | soon | never | days:30

Locally: npx swapvera-scan .

Test cases

One JSON object per line. input is a shortcut for one user message; expect holds the hard checks.

{"id": "refund-json", "input": "Return JSON for order 991", "expect": {"json_schema": {"type": "object", "required": ["order_id"]}}}

Hard checks: json, json_schema, contains, contains_any, not_contains, regex, max_chars, equals, tool, tool_args, no_tool.

Before you build a suite, swapvera predict shows what broke for other teams on the same migration, for the kinds of cases you have. See the migration atlas.

No cases yet: swapvera suite from-traces logs.jsonl converts OpenAI, Anthropic or Langfuse logs, and swapvera suite synth --system system_prompt.txt generates realistic inputs.

Verdict on every pull request

name: Swapvera verdict
on:
  pull_request:
    paths: ['swapvera.yaml', 'cases.jsonl', 'prompts/**']
permissions: { contents: read, pull-requests: write }
jobs:
  verdict:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: pixelric-spec/swapvera-verdict@v1
        with:
          license: ${{ secrets.SWAPVERA_LICENSE }}
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

Build it yourself or use Swapvera

You can write this yourself, and with a coding agent the first version takes an afternoon. A small script is enough when you have a few fixed cases and only hard checks, such as valid JSON or the right tool call. Use it and move on.

It gets harder when the answer depends on quality and you have to stand behind “safe to switch”. These are the parts a quick script usually gets wrong:

• Stopping when the result looks clear. Re-checking a classical test after every run took false alarms from 5% to 12% in our simulation. Swapvera uses tests that stay valid when they stop early, and keeps false discoveries at 5% across cases.
• Trusting the judge. An LLM judge can miss the defect that matters in your task. Swapvera damages your own answers on purpose before each verdict and checks that the judge notices. If it does not, a more capable judge takes over, and an uncertified judge cannot support “safe”.
• Judge bias. A judge can favor its own model family or whichever answer it read first. Swapvera swaps the order when it needs to, picks a judge from another family when one is available, and says so in the report when it cannot.
• Writing the test cases. Swapvera builds cases from your real traffic and derives checks from what your answers always do, so nobody has to write or label them.
• Saying too much from too little. With few cases, the honest answer is often “inconclusive”. A script that picks a winner hides that.
• Keeping it up to date. Shutdown dates, replacements and prices change every few months, and someone has to maintain them.

These results come from simulations with simulated providers and judges. The judge certificate has not been validated with human raters yet, and a verdict covers only the cases compared. How the verdict is decided.

How the verdict is decided

Hard checks decide first. If a check your old model passed now fails, that repetition counts against the new model, whatever the judge says.

A judge checked against order and family bias. Today our judges run on Claude. For models from other providers that is a different family. Between two Claude models it is the same family, and it can even be one of the two models: the report says so, and your hard checks still decide first. A pair is judged again with the order swapped when the first answer is not a tie, plus a 20% sample. If the answer flips, it counts as a tie, and the share of flips is reported as the judge’s consistency. Below 60%, the verdict cannot be “safe”.

Sequential testing that stays valid when it stops early. Every case runs twice on both models. Cases where the models behave identically stop there. Cases that differ keep running, and cases that look like regressions keep running until the evidence is decisive, up to 20 times. Per case, the evidence is an e-value from a test martingale, which stays valid under this kind of early stopping; a classical test re-checked after every run more than doubles its false alarms (12% instead of 5% in our simulation). Across cases, e-BH keeps the false discovery rate at 5% whatever the dependence between cases.

Overall effect with finite-sample intervals. The overall change uses only the first two runs of every case, so the adaptive extra runs cannot bias it, and confidence intervals built by betting (Waudby-Smith and Ramdas), which hold at any sample size instead of relying on large-sample approximations.

Optional: calibrate the judge against people. swapvera run --pairs pairs.jsonl, then swapvera labels pairs.jsonl writes a blind sample for you to label, and swapvera calibrate pairs.jsonl labels.jsonl recomputes the verdict with prediction-powered inference (PPI++), which corrects the judge’s bias using your labels.

Safe to migrate means: no case got significantly worse, the judge was certified and consistent, and the 95% intervals rule out a drop larger than 10 points in hard checks and 0.10 in net preference. When answers vary from run to run, that takes about 150 cases: in our A/A simulations, with 50 cases an identical model came out "safe" only 4% of the time, and 70% with 150. False alarms stayed at 0.5%. Do not migrate means the new model clearly fails required checks or clearly loses overall. A verdict covers only the cases in your suite.

Privacy

Dashboard (connected). The SDK sends a sample of your calls, prompts and answers included, because comparisons replay your real requests. They are stored in the EU (Supabase, Frankfurt), never used for training or shared, and deleted after 30 days by default (1 to 90, in the dashboard). Your provider key is encrypted (AES-GCM) and used only for comparisons you start.

CLI. Generations run from your machine or runner with your keys and never pass through us. The hosted judge receives the two answers and the conversation context for each comparison, returns a preference and stores nothing but a call count. Prefer to keep everything in-house: set judge.mode: byok.

This website. We count page views and clicks ourselves, with no cookies and no third-party analytics. Your IP address is not stored: visitors are counted with a hash that changes every day. Browsers with Do Not Track send nothing.

The migration atlas is opt-in. With --share (or share_anonymous_stats: true) and a license, the engine sends closed categories per case (output format, single or multi-turn, input size band, check types, whether it got worse and which check types started failing), the model pair and the overall verdict. Never prompts, outputs, case ids or check text, and never fine-tuned or self-hosted model names. The exact body is written to swapvera-atlas.json. A migration is published only after 5 different teams have shared it. Full list.