Conformance kit and fixtures
The spec files and fixtures the code is tested against, what each set holds, and which conformance test loads it, generated from test/conformance
What you'll learn
You're changing the spec, a schema, a transcript, or a binding's fixtures and need to know which test will hold you to it.
- what each spec file set holds
- which conformance test loads each set
- what each conformance test checks
The files under spec/ and test/fixtures/ that a test holds the code to, and the 25 tests in test/conformance that load them. go test ./test/conformance/ runs them alone, and make check runs them with the rest of the suite.
File sets
What each set holds comes from spec/README.md, or from the schema's title. A set is listed under every test whose code, or a helper it calls, reads it.
| Files | Holds | Loaded by |
|---|---|---|
spec/schemas/msg-*.schema.json (13: msg-attest.schema.json, msg-cost.schema.json, msg-end.schema.json, msg-error.schema.json, msg-log.schema.json, msg-open.schema.json, msg-pending.schema.json, msg-preflight.schema.json, msg-record-in.schema.json, msg-record-out.schema.json, msg-schema.schema.json, msg-state.schema.json, msg-verdict.schema.json) | Every wire-protocol message type, per direction | TestWireTranscriptValidatesAgainstSchemas, TestTranscriptRoundTripsThroughProtocol, TestEveryProtocolSchemaIsCompilable, TestCostSchemaAcceptsBasis, TestErrorSchema, TestOpenSchemaAcceptsAccepts |
spec/wire/*.ndjson (1: basic-run.ndjson) | Golden transcripts recorded from the real adapters | TestWireTranscriptValidatesAgainstSchemas, TestTranscriptRoundTripsThroughProtocol |
spec/schemas/pipeline.schema.json | pipeline.yaml, including uses:/variables:/on_missing: | TestSpecExamplePipelineValidates, TestExamplePipelinesValidate, TestEveryProtocolSchemaIsCompilable |
spec/schemas/manifest.schema.json | An adapter's manifest.json | TestAdapterManifestsValidate, TestEveryProtocolSchemaIsCompilable |
spec/schemas/field-registry.schema.json | The fields/*.json type files themselves | TestEveryProtocolSchemaIsCompilable, TestRegistryFilesValidateAgainstSchema |
spec/fields/*.json (3: company.json, person.json, post.json) | A type: its kind (subject or signal), its ordered identity key tiers, and its canonical fields — names, tiers, normalization rules, value domains, and any reference that mints a relation | TestRegistryFilesValidateAgainstSchema, TestEmbeddedRegistryMatchesSpecDir |
spec/ledger.sql | The canonical ledger DDL, including the current_fields view | TestLedgerSchemaMatchesSpec |
spec/acceptance/*.yaml (8: guard.yaml, interrogate.yaml, iterate.yaml, launch.yaml, recover.yaml, report.yaml, segment.yaml, top-up.yaml) | The eight operator stories as structured given/when/then | TestAcceptanceCorpusIsCompleteAndWellFormed |
spec/bindings/*/binding.yaml (1: apollo-search/binding.yaml) | — | TestShippedBindingsParse |
spec/bindings/*/fixtures/* (1: apollo-search/fixtures/conformance.json) | — | TestShippedBindingsParse |
spec/bundle-manifest.json | Campaign bundle manifest | TestBundleManifestsValidate |
test/fixtures/registry/*/binding.yaml (5: apollo-enrich/binding.yaml, apollo-search/binding.yaml, attio-assert/binding.yaml, harvest-profile/binding.yaml, harvest-recent-posts/binding.yaml) | — | TestShippedBindingsParse, TestApolloBindingConformance, TestApolloEnrichConformance, TestHarvestBindingConformance, TestAttioBindingConformance, TestSimulateGap, TestSimulateServesFixtures, TestHarvestRecentPostsConformance |
Tests
What each test checks is its doc comment.
| Test | File | Kit files it reads | Checks |
|---|---|---|---|
TestAcceptanceCorpusIsCompleteAndWellFormed | acceptance_test.go | spec/acceptance/*.yaml | TestAcceptanceCorpusIsCompleteAndWellFormed is a structural check only: it says the acceptance criteria are loadable, complete, and non-empty. It does not assert that any e2e test implements a story. |
TestShippedBindingsParse | binding_test.go | spec/bindings/*/binding.yaml, spec/bindings/*/fixtures/*, test/fixtures/registry/*/binding.yaml | The one binding under spec/bindings/ (the worked example) and every registry entry in the local copy conform to the schema, bridge onto a valid §6 manifest, and name only registry-valid fields in their extraction (§4a enforcement extended to bindings). |
TestApolloBindingConformance | binding_test.go | test/fixtures/registry/*/binding.yaml | — |
TestApolloEnrichConformance | binding_test.go | test/fixtures/registry/*/binding.yaml | The paid half of the split (ADR-043) — one masked record in, the revealed surface out, one per-credit cost row. |
TestHarvestBindingConformance | binding_test.go | test/fixtures/registry/*/binding.yaml | harvest/profile@2 (ADR-059) — one call, role_history through each:, the cost from config at its default. |
TestAttioBindingConformance | binding_test.go | test/fixtures/registry/*/binding.yaml | — |
TestSimulateGap | binding_test.go | test/fixtures/registry/*/binding.yaml | A binding without fixtures, asked to simulate, surfaces the gap instead of touching the network or silently passing (SPEC §8). |
TestSimulateServesFixtures | binding_test.go | test/fixtures/registry/*/binding.yaml | With fixtures present and GTME_SIMULATE set, the engine answers from fixtures without any HTTP seam at all. |
TestHarvestRecentPostsConformance | binding_test.go | test/fixtures/registry/*/binding.yaml | harvest/recent-posts keeps at most posts_limit non-empty posts, content before text (ADR-059). |
TestLedgerSchemaMatchesSpec | ledger_schema_test.go | spec/ledger.sql | TestLedgerSchemaMatchesSpec is the ADR-010 schema-conformance test: the canonical DDL in spec/ledger.sql and the schema the migrations actually build must describe the same ledger. |
TestSpecExamplePipelineValidates | pipeline_test.go | spec/schemas/pipeline.schema.json | TestSpecExamplePipelineValidates checks SPEC §9's own example pipeline against spec/schemas/pipeline.schema.json. The example includes uses: (ADR-004). |
TestSpecExamplePipelineParses | pipeline_test.go | — | TestSpecExamplePipelineParses is the other half of the §9 contract: the schema saying a document is well-formed is worth nothing if the parser the operator actually hits rejects it. SPEC §9's example carries uses: (ADR-004), so this fails until internal/pipeline knows that key. |
TestExamplePipelinesValidate | pipeline_test.go | spec/schemas/pipeline.schema.json | TestExamplePipelinesValidate checks every pipeline shipped under examples/. |
TestAdapterManifestsValidate | pipeline_test.go | spec/schemas/manifest.schema.json | TestAdapterManifestsValidate checks every manifest.json in the tree against spec/schemas/manifest.schema.json — built-ins, the external Python adapter, and the test fixture adapters alike, since SPEC §6 makes no distinction. |
TestBundleManifestsValidate | pipeline_test.go | spec/bundle-manifest.json | TestBundleManifestsValidate checks every frozen pattern bundle's manifest.json under bundles/ against spec/bundle-manifest.json (SPEC §8, ADR-029) — the format the bundler writes and gtme run \<bundle\> reads. |
TestPluginManifestsAreWellFormed | pipeline_test.go | — | The Claude Code plugin ships from this repo (repo root is the marketplace, plugin/ is the plugin). The two manifests must parse, name what the docs say they name, and the plugin's version must match the binary's tag line — both bump in one commit. |
TestWireTranscriptValidatesAgainstSchemas | protocol_test.go | spec/schemas/msg-*.schema.json, spec/wire/*.ndjson | TestWireTranscriptValidatesAgainstSchemas is the ADR-010 protocol-conformance test: every message in every golden transcript must validate against the spec/schemas/ file for its type and direction. |
TestTranscriptRoundTripsThroughProtocol | protocol_test.go | spec/schemas/msg-*.schema.json, spec/wire/*.ndjson | TestTranscriptRoundTripsThroughProtocol checks the other half of the contract: the golden lines are not just schema-valid, they are what internal/protocol actually reads and writes. A message that changes shape on the way through is a serialization divergence. |
TestEveryProtocolSchemaIsCompilable | protocol_test.go | spec/schemas/msg-*.schema.json, spec/schemas/pipeline.schema.json, spec/schemas/manifest.schema.json, spec/schemas/field-registry.schema.json | TestEveryProtocolSchemaIsCompilable guards the corpus itself: a schema nobody happens to exercise still has to be loadable draft-07. |
TestCostSchemaAcceptsBasis | protocol_test.go | spec/schemas/msg-*.schema.json | The COST schema admits ADR-046's basis and nothing else in that slot — a labeled emission from a built-in must validate, and a made-up basis must not. |
TestErrorSchema | protocol_test.go | spec/schemas/msg-*.schema.json | ADR-065's ERROR carries the binding verdict vocabulary. A keyed fail_record and a keyless fail_run validate; a verdict outside the four, a missing reason, or a fail_record without a key does not (SPEC §5). |
TestOpenSchemaAcceptsAccepts | protocol_test.go | spec/schemas/msg-*.schema.json | OPEN's optional accepts lists the optional messages a runner acts on (ADR-065). |
TestRegistryFilesValidateAgainstSchema | registry_test.go | spec/schemas/field-registry.schema.json, spec/fields/*.json | — |
TestEmbeddedRegistryMatchesSpecDir | registry_test.go | spec/fields/*.json | The embedded registry is the same files — the binary can never disagree with the spec/ directory it was built from. |
TestShippedManifestsPassRegistryValidation | registry_test.go | — | Enforcement layer 1 over everything this repo ships: every property named in a manifest's static needs/provides schemas is canonical for its entity type or vendor-namespaced (SPEC §4a). |
spec/wire/README.md says how a wire transcript is recorded and which schema each of its lines must meet.