> ## Documentation Index > Fetch the complete documentation index at: https://docs.deploybaseline.com/llms.txt > Use this file to discover all available pages before exploring further. # Testing and conformance > The kinds of tests Baseline runs, the gates that keep providers honest, how to run each one, and what CI enforces. Baseline's tests answer two questions: does the emulator do what its own tests say, and does it match the real vendor API. The second is what [CONFORMANCE.md](https://github.com/deploybaseline/cli/blob/main/CONFORMANCE.md) reports per provider. ## At a glance | Check | Lives in | Runs with | In CI | | - | - | - | - | | SDK tests | `src/__tests__/*.test.ts` | `make test-provider P=` | Every push | | Coverage gate 1 (matrix vs routes) | `coverage.test.ts` | Any test run | Every push | | Coverage gate 2 (every route hit) | `coverage.setup.ts` | Full provider run | Every push | | Spec gate | `coverage.setup.ts`, `spec-allow.ts` | Full provider run | Every push | | Pinned gaps | `it.fails` tests, `gaps` in the matrix | Any test run | Every push | | Property and model tests | `*.property.test.ts`, `*.model.test.ts` | Any test run | Every push | | State restart | `state-restart*.test.ts` | Any test run | Every push | | Saved-state fixtures | `packages/cli/src/__tests__/state-fixtures.test.ts` | Any test run | Every push | | Provider compatibility | `scripts/compatibility.mjs` | `make compatibility` | Every push (packed), daily (published) | | Provider packages | `scripts/third-party-acceptance.mjs`, `examples/provider-template` | `make third-party` | Every push | | New install | `scripts/slim-cli-acceptance.mjs` | `make slim-cli` | Every push | | Persistence contract | `tests/contracts/persistence.js` | CLI and adapter tests | Every push | | Custom consumer acceptance | `scripts/custom-api-acceptance.mjs` | `pnpm test:custom-consumer` | Every push, 3 OSes | | Dual mode | `*.dual.test.ts` | `pnpm --filter test:dual` | Manual and monthly | ## SDK tests Providers are tested by driving them with the vendor's official SDK, pinned to an exact version in `devDependencies` (for example `"resend": "6.34.1"`). `startService` from `@deploybaseline/testing` starts the plugin in process on a free port and returns its `url` for the SDK's base URL option: ```ts theme={null} const service = await startService(resendPlugin, { seedFromConfig, seed: { domains: [{ name: "seeded.example.com" }] }, tokens: { [TOKEN]: { login: "sdk@example.com" } }, }); const resend = new Resend(TOKEN, { baseUrl: service.url }); ``` Call `closeAllServices` in `afterAll` or `afterEach`. `createWebhookSink` gives you a local receiver for webhook assertions. Each package's `vitest.config.ts` spreads `providerTestConfig` from `@deploybaseline/testing/vitest`, which sets a 10 second timeout, excludes `*.dual.test.ts`, points core at the cache directories the gates read, and reuses workers across test files with a per-file setup that undoes spies, stubbed globals and env, and fake timers. See [How tests share the machine](/contributing/local-development.md#how-tests-share-the-machine). ## Coverage matrix `src/__tests__/coverage.ts` lists every route the plugin registers as `"METHOD /pattern"` (or `WS /pattern`, or a GraphQL root field such as `Query.issues`), with a status: * `supported`: works like the real API for the flows we know about * `partial`: works, with known gaps described in `notes` or pinned by `gaps` * `not_started`: planned, not registered yet Refresh it from the built plugin with: ```bash theme={null} pnpm --filter @deploybaseline/ build node scripts/generate-coverage-matrix.mjs # write node scripts/generate-coverage-matrix.mjs --check # print what would change ``` Existing entries keep their fields. New routes become `supported`, and entries the last full run did not hit get the note "No test yet". **Gate 1: matrix matches routes** `coverage.test.ts` calls `checkRouteCoverage(coverage, pluginRoutes(plugin))`. It fails when a registered route is missing from the matrix or the matrix lists a route that is not registered. Composed services use `checkModuleCoverage` so each route sits under the module that registered it. **Gate 2: every route is hit** `coverage.setup.ts` exports `coverageGate({ plugin, coverage, mode })` as a Vitest global setup. While tests run, core's router records each matched route. After a full run the gate fails (`mode: "fail"`) or warns (`mode: "warn"`, with an optional `failFrom` date) when a `supported` or `partial` route was never hit. Filtered runs (`-t`, a single file, watch mode) skip the gate, so local iteration is never blocked. In CI each gate appends a row to the job summary's "Route coverage" table. ## Spec gate When a vendor publishes a usable OpenAPI document, its trimmed copy lives at `spec/openapi.json` (see [Vendored specs](/contributing/docs-and-api-reference.md#vendored-specs)). During a full run, core samples JSON responses per route and status, and the gate validates each against the operation's response schema, naming the operation and JSON path of every mismatch. **1. Turn it on in warn mode** Add `spec: { mode: "warn" }` to `coverageGate` in `coverage.setup.ts` and run the full suite. **2. Record the current mismatches** ```bash theme={null} pnpm --filter @deploybaseline/ test node scripts/generate-spec-allow.mjs ``` This writes `src/__tests__/spec-allow.ts`. Existing entries keep their reason; entries that no longer match are dropped. **3. Triage and switch to fail** Fix what you can, give every remaining entry a specific reason, then set `mode: "fail"`. In fail mode an allow entry that matches nothing also fails, so the list cannot rot. Options: `successOnly: true` when the vendor spec documents only successful responses (Twilio, Resend), so undocumented error statuses are not reported; `specPath` when the emulator's route pattern is spelled differently from the vendor's path. ## Pinned gaps Pin a known difference from the real API as an `it.fails` test, so fixing the gap flips the test. List the test's name in the `gaps` of its matrix entry and mark the entry `partial`. `checkPinnedGaps` (called from `coverage.test.ts`) keeps the two in sync. ## Property and model tests * Parsers get `fast-check` property tests: `*.property.test.ts` (for example `stripe/src/__tests__/form-body.property.test.ts`). * State machines get a model-based test: `*.model.test.ts` runs random command sequences against a small reference model (Stripe has several, such as `subscriptions.model.test.ts`). Commit any counterexample they find as a regular test. ## State restart and persistence * Every provider has a `state-restart` test that runs a main flow, calls `restartService` (save, then start a fresh instance from the saved JSON on the same port), and checks that reads match and tokens still work. * `tests/contracts/persistence.js` is one shared contract for saved state: a restart keeps writes, tokens, and signing keys; a `stateVersion` mismatch is refused and left alone; reset; and the single-writer lock. It runs from `persistence-contract.test.ts` in `packages/cli` against both `createEmulator` and the `start` runner. A change to how either runtime saves or loads state must keep it passing. * `state-fixtures.test.ts` restores the state each earlier provider version saved (`packages/cli/src/__tests__/fixtures/state//.json`) and checks that the same state under another `stateVersion` is refused. After bumping a provider, `pnpm state-fixtures` adds the new version's fixture. ## Custom consumer acceptance ```bash theme={null} pnpm --filter "@deploybaseline/cli..." build pnpm test:custom-consumer ``` `scripts/custom-api-acceptance.mjs` installs the packed CLI into a fresh temporary project and repeats the CLI checks there, including a hard kill and stale lock takeover. CI runs it on Linux, macOS, and Windows with Node 24, and on Linux with Node 26. ## Dual mode Dual mode runs the same SDK scenarios against the emulator and the vendor's sandbox, then compares response shapes. It is off unless `DUAL_MODE` names the provider, refuses anything that is not a sandbox credential, and tags what it creates so leftovers can be swept. | Provider | Environment variables | Sandbox requirement | | - | - | - | | `stripe` | `STRIPE_TEST_KEY=sk_test_...` | Test mode; creates resources tagged `baseline_dual=1` | | `twilio` | `TWILIO_TEST_ACCOUNT_SID`, `TWILIO_TEST_AUTH_TOKEN` | Test credentials; creates nothing | | `clerk` | `CLERK_TEST_SECRET_KEY=sk_test_...` | Development instance with Organizations on | | `okta` | `OKTA_TEST_ORG_URL`, `OKTA_TEST_API_TOKEN` | Developer, trial, or preview org with a `baseline-dual-sandbox` group | | `github` | `GH_TEST_ORG`, `GH_TEST_TOKEN` | Org with a `baseline-dual-sandbox` repository | | `slack` | `SLACK_TEST_BOT_TOKEN=xoxb-...` | Workspace with a public `#baseline-dual-sandbox` channel | | `linear` | `LINEAR_TEST_API_KEY=lin_api_...` | Workspace with a `baseline-dual-sandbox` team | ```bash theme={null} DUAL_MODE=stripe STRIPE_TEST_KEY=sk_test_... pnpm --filter @deploybaseline/stripe test:dual STRIPE_TEST_KEY=sk_test_... node scripts/dual-cleanup.mjs stripe --dry-run ``` `node scripts/dual-cleanup.mjs ` sweeps leftovers (add `--dry-run` to preview). The `Dual mode` workflow runs one provider on demand and all of them monthly from the protected `dual-mode` environment, uploads a comparison report, and on success opens a PR that records the date in `conformance/verified.json`. > [!WARNING] > > Never point dual mode at production credentials. The guards in `@deploybaseline/testing/src/dual.ts` reject live keys, but use dedicated sandbox accounts regardless. ## Conformance report ```bash theme={null} make conformance # pnpm conformance ``` `scripts/conformance.mjs` regenerates `CONFORMANCE.md` and `docs/snippets/conformance/.mdx` from repository content only (matrices, gate settings, vendored specs, the lockfile, `conformance/verified.json`), so the output is stable and CI can diff it. ## What CI enforces The `build-and-test` job in `.github/workflows/ci.yml` runs, in order: **1. Install and build** `pnpm install --frozen-lockfile`, then `pnpm build`. **2. Bundle budget** `node scripts/check-bundle.mjs`: CLI tarball size, the code `--help` loads, and that the build inlines no provider (from `dist/metafile-esm.json`). **3. Packaged notices** Every packed package carries the root `LICENSE`, `NOTICE`, and `THIRD_PARTY_NOTICES.md`. **4. Release plan** `node scripts/release-packages.mjs plan --dry-run`: which package versions a merge would publish, with a warning for any that has no changelog entry. **5. Packed providers** `node scripts/compatibility.mjs host --packed`: the built CLI starts every provider from its packed tarball, installed into an empty project. **6. Provider packages** `pnpm test:third-party`: the provider template, packed and served from a local registry, is refused until trusted, pinned by `trust`, run next to Stripe, and refused again when `baseline.lock` names another integrity. **7. New install** `pnpm test:slim-cli`: the packed CLI, installed with npm into an empty project, holds no provider code; a first start with an empty cache pulls the default providers from a local registry, a second runs them from the cache, and an offline start with no cache says to run `pull`. **8. Generated files are fresh** `pnpm conformance` followed by `git diff --exit-code` on `CONFORMANCE.md` and `docs/snippets/conformance`, and `node scripts/generate-api-reference.mjs --check` for the API reference. **9. Static checks** `pnpm lint` (Biome lint and format check), `pnpm type-check`. **10. Tests** `pnpm test`, with every coverage and spec gate. Alongside it, the `Docs (Mintlify)` job runs `mint validate` and `mint broken-links`, and the `custom-consumer` matrix runs `pnpm test:custom-consumer`. `make check` runs the same set locally. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.