pipeline.yaml keys

Every key a pipeline file accepts, top level and per step, with its type and whether it's required, generated from spec/schemas/pipeline.schema.json

What you'll learn

You're writing or reading a pipeline file and need the exact key, what it takes, and where it's allowed.

every top-level key and every step key
which keys are required and what type each takes
a whole pipeline file that uses them

Every key spec/schemas/pipeline.schema.json allows in a pipeline file: the top-level keys first, then the keys of a step, which source: and every entry of steps: share. A dotted key sits inside the key before the dot, and an upper-case part such as FIELD stands for a name you choose.

The only pipeline authoring surface in v0 (SPEC.md §9, ADR-005). Validate the YAML document after decoding it to JSON. Derived from internal/pipeline.Pipeline, which decodes with KnownFields(true) — hence additionalProperties: false.

Top-level keys

Key	Type	Required	Description
`name`	string	yes	Pipeline name; recorded in runs.pipeline.
`version`	always `1`	no	Format version. Defaults to 1 when omitted; v0 understands only 1.
`source`	step (see Step keys)	yes	The one source step. Required. `when:` is not allowed on it.
`steps`	array of step (see Step keys)	no	The pipeline's steps, executed strictly in order. Deliver-role adapters are ordinary entries (ADR-031) — any number, any position; there is no top-level deliver block.
`group`	string	no	ADR-021: the membership terminus. Records completing the run's final step are `added` to this group (created on demand). Valid with or without deliver steps; captures completers, not sends (ADR-031).
`waterfall`	any	no	Reserved (SPEC.md §9). Present only so v1 can add it without a format break; any value here MUST be rejected in v0.

Step keys

Key	Type	Required	Description
`id`	string	no	Step id, unique within the pipeline. Defaults to 'source' / 'step-N'.
`use`	string	one of `use`, `group`	Adapter id, such as 'harvest/profile'. Required everywhere except a group source, which declares group: instead.
`group`	string	one of `use`, `group`	ADR-021: group-as-source, valid only on the source step and mutually exclusive with use:. Members are projected from the ledger; provides are open (wildcard), needs enforced per record at run time.
`require`	array of string	no	ADR-021: process only records currently members of EVERY listed group. Valid on any non-source step, deliver steps included. Referenced groups must exist at plan time.
`exclude`	array of string	no	ADR-021: skip records currently members of ANY listed group — the judgment-memory mechanism. Valid on any non-source step, deliver steps included. Referenced groups must exist at plan time.
`record`	string	no	ADR-021/031, deliver steps only: the group this step's successful deliveries append `touched` events to. Defaults to the pipeline name per deliver step (steps sharing the default share the scope); created on demand.
`suppress`	object	no	ADR-021/031, deliver steps only: skip records with a `touched` event in the group within the window; skips are receipted with reasons and the record advances past the step. Layers above the idempotency floor.
`suppress.group`	string	yes	—
`suppress.within`	string matching `^[0-9]+d$`	yes	—
`with`	object	no	The step's config, validated against the adapter's config_schema. Sent as OPEN.config. On a participant step (ai/, human/, agent/, text/) it MAY carry template: (ADR-057); on a human/* step render: and prompt: (ADR-049); engine: is not a key anywhere (ADR-050).
`with.template`	string or object	no	ADR-057, participant steps: the operator's text, a string or {file: <path>} (relative to the pipeline file), in the bounded Liquid dialect of SPEC.md §10 item 10. An ai/* step renders it once over config.* (the shared block; records never enter it); a text/, human/ or agent/* step renders it per record over config.* and record., the latter limited to uses:/of:. The planner parses it and refuses an unknown tag or filter, a record. reference out of scope, or a config.* key absent from with:.
`with.template.file`	string	yes	—
`with.render`	object	no	ADR-049, human/* and agent/* steps: the fields the participant is shown per record (validated as uses:, SPEC.md §7). Default: the uses: fields, or the of: value alone; a template: (ADR-057) replaces the listing.
`with.render.fields`	array of string	no	—
`with.prompt`	one of `tty`, `never`	no	On a human/* step (ADR-049): tty (default) walks the records inside `gtme run` at a terminal, never leaves every record pending for `gtme answer`; an agent/* step is always never. Never text: an ai/* step's text is template: (ADR-057), and with.prompt there is a plan error naming it.
`with.engine`	not allowed	no	Removed (ADR-050): the API is the only model engine and the fixture engine is selected by environment. A step carrying engine: fails `gtme plan` naming the fix; `engine: claude-code` becomes an agent/* step answered through `gtme answer`.
`uses`	array of string	no	ADR-004: a participant step's dynamic needs. Valid only on steps whose adapter role is filter/compose/review (the participant roles, ADR-048: ai/, human/, agent/, and text/, ADR-057). The planner MUST validate it exactly as needs.required (SPEC.md §7), and at execution time the projected fields object MUST be built from `uses` rather than from the manifest's open-ended `needs`.
`of`	string	no	ADR-048: the referent — the field whose value a compose or review step is about (required on a review; an edit on a compose). Validated exactly as a uses: entry (SPEC.md §7); its current value joins the record's judgment-cache input hash, and its field_values.id is recorded as the referent of everything the step writes.
`when`	string matching `^[A-Za-z0-9_.\-]+\.passed$`	no	Gate. v0 supports only '<step_id>.passed', referencing an earlier step.
`cache`	string matching `^[0-9]+d$`	no	Freshness-window override for this step, such as '30d'.
`idempotency`	string	no	Deliver steps only (ADR-031: the planner rejects it elsewhere): the field whose value keys a delivery. Defaults to the identity key (SPEC.md §8); REQUIRED on http/deliver (ADR-023).
`variables`	map of string	no	Deliver steps only (ADR-018/019/031): egress mapping of target merge-field name → canonical or namespaced ledger field. Its values are the step's dynamic needs (SPEC.md §6, §7); the planner validates each against upstream provides and rejects the key on any other step.
`redeliver`	one of `always`, `on_change`, `never`	no	ADR-045, deliver steps only: repeat policy. Defaults to on_change when the adapter declares idempotency: native, never otherwise; always/on_change are plan errors on a target without native idempotency.
`on_missing`	one of `run`, `skip`, `fail`	no	Per-record completeness policy. Deliver steps (ADR-019/031, SPEC.md §8): skip (default) or fail when a variables: target does not resolve to a non-empty value — skip records a verdict and the record advances past the step; fail freezes it; blank merge fields never send, so run is not valid here. Participant steps (ai/, human/, agent/*; ADR-053, SPEC.md §7): run (default) dispatches the record with a declared uses: field absent and the receipt counts it; skip advances the record untouched without dispatching; fail fails it naming the missing fields.
`waterfall`	any	no	Reserved (SPEC.md §9); rejected in v0.
`provides`	array of string or object	no	ADR-033: declared output fields, valid only on participant steps (ai/, human/, agent/*; ADR-048). A review MUST declare its labels here. A list of field names, or an object whose keys are field names and whose values are null or declare type, enum, canonical. Bare names land as <pipeline>.<name>; a dotted name is kept as written; canonical: true lands the field on the canonical field of that name (SPEC.md §7).
`provides.FIELD`	null or object	no	—
`provides.FIELD.type`	one of `string`, `integer`, `number`, `boolean`, `array`	no	—
`provides.FIELD.enum`	array of string	no	—
`provides.FIELD.canonical`	boolean	no	SPEC.md §7: the declared name IS a canonical field of the pipeline's entity type; the output lands there, global rather than per-campaign.
`limit`	integer	no	ADR-032: on a group source, at most N current members, oldest-added first. ADR-054: on a traverse step, the engine-owned cap on children per parent, as on a source binding. Valid nowhere else.
`once`	boolean	no	ADR-052: group source only — select only members this pipeline has not already finished (completed the final step, or stopped by a filter verdict); failed and pending records stay eligible. Oldest-added first within limit. Never mutates the group; a dry or simulated run finishes nothing.
`respend`	boolean	no	ADR-038: declares that re-running the pipeline MAY pay for this step's records again — silences the plan's respend warning (SPEC.md §7). Not valid on the source.

Example

From examples/hello.yaml: a source and 3 steps, score, keep, and out.

# See it run's second file (START.md): the top-up receipt with zero keys, on a
# ledger that persists. demo/enrich is the binary's own synthetic enrichment
# (SPEC §10 item 9): no vendor, no key, no network — deterministic scores,
# labelled synthetic in the value itself, at a stated pretend price of
# $0.01 per record so the receipt's arithmetic is real.
#
#   gtme run examples/hello.yaml      # 3 scored at $0.01 each; 1 kept; out.csv written
#   gtme run examples/hello.yaml      # again: 3 cached, $0.03 avoided, nothing delivered twice
#
# Every dollar it prints is labelled demo/enrich wherever it appears.
name: hello
version: 1

source:
  use: csv/source
  with:
    path: contacts.csv          # three fictional people, beside this file
    columns: { full_name: Full Name, email: Email, title: Title, company_domain: Company Website }

steps:
  - id: score
    use: demo/enrich            # $0.01 per record, pretend; cached for 30 days
    with:
      cost_per_record_usd: 0.01

  - id: keep                    # deterministic judgment in SQL: no model, no spend
    use: sql/filter
    with:
      query: >
        SELECT identity_id FROM current_values
        WHERE field = 'demo.score' AND CAST(value AS INTEGER) >= 70

  - id: out
    use: csv/deliver
    with:
      path: out.csv
    variables:
      score: demo.score
      note: demo.note
    idempotency: email          # re-runs append nothing twice

See also

Used in

A pipeline is a YAML file: Every key, its type, and where it's valid
Groups and segments: The group keys with their types and where each is valid