Testing and conformance

Baseline’s tests answer two questions: does the emulator do what its own tests say, and does it match the real vendor API. The second is what CONFORMANCE.md reports per provider.

At a glance

Check	Lives in	Runs with	In CI
SDK tests	`src/__tests__/*.test.ts`	`make test-provider P=<name>`	Every push
Coverage gate 1 (matrix vs routes)	`coverage.test.ts`	Any test run	Every push
Coverage gate 2 (every route hit)	`coverage.setup.ts`	Full provider run	Every push
Spec gate	`coverage.setup.ts`, `spec-allow.ts`	Full provider run	Every push
Pinned gaps	`it.fails` tests, `gaps` in the matrix	Any test run	Every push
Property and model tests	`.property.test.ts`, `.model.test.ts`	Any test run	Every push
State restart	`state-restart*.test.ts`	Any test run	Every push
Saved-state fixtures	`packages/cli/src/__tests__/state-fixtures.test.ts`	Any test run	Every push
Provider compatibility	`scripts/compatibility.mjs`	`make compatibility`	Every push (packed), daily (published)
Provider packages	`scripts/third-party-acceptance.mjs`, `examples/provider-template`	`make third-party`	Every push
New install	`scripts/slim-cli-acceptance.mjs`	`make slim-cli`	Every push
Persistence contract	`tests/contracts/persistence.js`	CLI and adapter tests	Every push
Custom consumer acceptance	`scripts/custom-api-acceptance.mjs`	`pnpm test:custom-consumer`	Every push, 3 OSes
Dual mode	`*.dual.test.ts`	`pnpm --filter <pkg> test:dual`	Manual and monthly

SDK tests

Providers are tested by driving them with the vendor’s official SDK, pinned to an exact version in devDependencies (for example "resend": "6.34.1"). startService from @deploybaseline/testing starts the plugin in process on a free port and returns its url for the SDK’s base URL option:

const service = await startService<ResendSeedConfig>(resendPlugin, {
  seedFromConfig,
  seed: { domains: [{ name: "seeded.example.com" }] },
  tokens: { [TOKEN]: { login: "[email protected]" } },
});
const resend = new Resend(TOKEN, { baseUrl: service.url });

Call closeAllServices in afterAll or afterEach. createWebhookSink gives you a local receiver for webhook assertions. Each package’s vitest.config.ts spreads providerTestConfig from @deploybaseline/testing/vitest, which sets a 10 second timeout, excludes *.dual.test.ts, points core at the cache directories the gates read, and reuses workers across test files with a per-file setup that undoes spies, stubbed globals and env, and fake timers. See How tests share the machine.

Coverage matrix

src/__tests__/coverage.ts lists every route the plugin registers as "METHOD /pattern" (or WS /pattern, or a GraphQL root field such as Query.issues), with a status:

supported: works like the real API for the flows we know about
partial: works, with known gaps described in notes or pinned by gaps
not_started: planned, not registered yet

Refresh it from the built plugin with:

pnpm --filter @deploybaseline/<name> build
node scripts/generate-coverage-matrix.mjs <name>           # write
node scripts/generate-coverage-matrix.mjs <name> --check   # print what would change

Existing entries keep their fields. New routes become supported, and entries the last full run did not hit get the note “No test yet”.

Gate 1: matrix matches routes
Gate 2: every route is hit

coverage.test.ts calls checkRouteCoverage(coverage, pluginRoutes(plugin)). It fails when a registered route is missing from the matrix or the matrix lists a route that is not registered. Composed services use checkModuleCoverage so each route sits under the module that registered it.

coverage.setup.ts exports coverageGate({ plugin, coverage, mode }) as a Vitest global setup. While tests run, core’s router records each matched route. After a full run the gate fails (mode: "fail") or warns (mode: "warn", with an optional failFrom date) when a supported or partial route was never hit. Filtered runs (-t, a single file, watch mode) skip the gate, so local iteration is never blocked. In CI each gate appends a row to the job summary’s “Route coverage” table.

Spec gate

When a vendor publishes a usable OpenAPI document, its trimmed copy lives at spec/openapi.json (see Vendored specs). During a full run, core samples JSON responses per route and status, and the gate validates each against the operation’s response schema, naming the operation and JSON path of every mismatch.

Turn it on in warn mode

Add spec: { mode: "warn" } to coverageGate in coverage.setup.ts and run the full suite.

Record the current mismatches

pnpm --filter @deploybaseline/<name> test
node scripts/generate-spec-allow.mjs <name>

This writes src/__tests__/spec-allow.ts. Existing entries keep their reason; entries that no longer match are dropped.

Triage and switch to fail

Fix what you can, give every remaining entry a specific reason, then set mode: "fail". In fail mode an allow entry that matches nothing also fails, so the list cannot rot.

Options: successOnly: true when the vendor spec documents only successful responses (Twilio, Resend), so undocumented error statuses are not reported; specPath when the emulator’s route pattern is spelled differently from the vendor’s path.

Pinned gaps

Pin a known difference from the real API as an it.fails test, so fixing the gap flips the test. List the test’s name in the gaps of its matrix entry and mark the entry partial. checkPinnedGaps (called from coverage.test.ts) keeps the two in sync.

Property and model tests

Parsers get fast-check property tests: *.property.test.ts (for example stripe/src/__tests__/form-body.property.test.ts).
State machines get a model-based test: *.model.test.ts runs random command sequences against a small reference model (Stripe has several, such as subscriptions.model.test.ts).

Commit any counterexample they find as a regular test.

State restart and persistence

Every provider has a state-restart test that runs a main flow, calls restartService (save, then start a fresh instance from the saved JSON on the same port), and checks that reads match and tokens still work.
tests/contracts/persistence.js is one shared contract for saved state: a restart keeps writes, tokens, and signing keys; a stateVersion mismatch is refused and left alone; reset; and the single-writer lock. It runs from persistence-contract.test.ts in packages/cli against both createEmulator and the start runner. A change to how either runtime saves or loads state must keep it passing.
state-fixtures.test.ts restores the state each earlier provider version saved (packages/cli/src/__tests__/fixtures/state/<provider>/<version>.json) and checks that the same state under another stateVersion is refused. After bumping a provider, pnpm state-fixtures adds the new version’s fixture.

Custom consumer acceptance

pnpm --filter "@deploybaseline/cli..." build
pnpm test:custom-consumer

scripts/custom-api-acceptance.mjs installs the packed CLI into a fresh temporary project and repeats the CLI checks there, including a hard kill and stale lock takeover. CI runs it on Linux, macOS, and Windows with Node 24, and on Linux with Node 26.

Dual mode

Dual mode runs the same SDK scenarios against the emulator and the vendor’s sandbox, then compares response shapes. It is off unless DUAL_MODE names the provider, refuses anything that is not a sandbox credential, and tags what it creates so leftovers can be swept.

Provider	Environment variables	Sandbox requirement
`stripe`	`STRIPE_TEST_KEY=sk_test_...`	Test mode; creates resources tagged `baseline_dual=1`
`twilio`	`TWILIO_TEST_ACCOUNT_SID`, `TWILIO_TEST_AUTH_TOKEN`	Test credentials; creates nothing
`clerk`	`CLERK_TEST_SECRET_KEY=sk_test_...`	Development instance with Organizations on
`okta`	`OKTA_TEST_ORG_URL`, `OKTA_TEST_API_TOKEN`	Developer, trial, or preview org with a `baseline-dual-sandbox` group
`github`	`GH_TEST_ORG`, `GH_TEST_TOKEN`	Org with a `baseline-dual-sandbox` repository
`slack`	`SLACK_TEST_BOT_TOKEN=xoxb-...`	Workspace with a public `#baseline-dual-sandbox` channel
`linear`	`LINEAR_TEST_API_KEY=lin_api_...`	Workspace with a `baseline-dual-sandbox` team

DUAL_MODE=stripe STRIPE_TEST_KEY=sk_test_... pnpm --filter @deploybaseline/stripe test:dual
STRIPE_TEST_KEY=sk_test_... node scripts/dual-cleanup.mjs stripe --dry-run

node scripts/dual-cleanup.mjs <provider> sweeps leftovers (add --dry-run to preview). The Dual mode workflow runs one provider on demand and all of them monthly from the protected dual-mode environment, uploads a comparison report, and on success opens a PR that records the date in conformance/verified.json.

Never point dual mode at production credentials. The guards in @deploybaseline/testing/src/dual.ts reject live keys, but use dedicated sandbox accounts regardless.

Conformance report

make conformance   # pnpm conformance

scripts/conformance.mjs regenerates CONFORMANCE.md and docs/snippets/conformance/<name>.mdx from repository content only (matrices, gate settings, vendored specs, the lockfile, conformance/verified.json), so the output is stable and CI can diff it.

What CI enforces

The build-and-test job in .github/workflows/ci.yml runs, in order:

Install and build

pnpm install --frozen-lockfile, then pnpm build.

Bundle budget

node scripts/check-bundle.mjs: CLI tarball size, the code --help loads, and that the build inlines no provider (from dist/metafile-esm.json).

Packaged notices

Every packed package carries the root LICENSE, NOTICE, and THIRD_PARTY_NOTICES.md.

Release plan

node scripts/release-packages.mjs plan --dry-run: which package versions a merge would publish, with a warning for any that has no changelog entry.

Packed providers

node scripts/compatibility.mjs host --packed: the built CLI starts every provider from its packed tarball, installed into an empty project.

Provider packages

pnpm test:third-party: the provider template, packed and served from a local registry, is refused until trusted, pinned by trust, run next to Stripe, and refused again when baseline.lock names another integrity.

New install

pnpm test:slim-cli: the packed CLI, installed with npm into an empty project, holds no provider code; a first start with an empty cache pulls the default providers from a local registry, a second runs them from the cache, and an offline start with no cache says to run pull.

Generated files are fresh

pnpm conformance followed by git diff --exit-code on CONFORMANCE.md and docs/snippets/conformance, and node scripts/generate-api-reference.mjs --check for the API reference.

Static checks

pnpm lint (Biome lint and format check), pnpm type-check.

Tests

pnpm test, with every coverage and spec gate.

Alongside it, the Docs (Mintlify) job runs mint validate and mint broken-links, and the custom-consumer matrix runs pnpm test:custom-consumer. make check runs the same set locally.

Contributing

Workflows

Testing and conformance

At a glance

SDK tests

Coverage matrix

Spec gate

Pinned gaps

Property and model tests

State restart and persistence

Custom consumer acceptance

Dual mode

Conformance report

What CI enforces

​At a glance

​SDK tests

​Coverage matrix

​Spec gate

​Pinned gaps

​Property and model tests

​State restart and persistence

​Custom consumer acceptance

​Dual mode

​Conformance report

​What CI enforces

At a glance

SDK tests

Coverage matrix

Spec gate

Pinned gaps

Property and model tests

State restart and persistence

Custom consumer acceptance

Dual mode

Conformance report

What CI enforces