Concepts

The ideas behind gtme in two parts, how a run works and how the model works, ordered so each page only needs the ones before it

What you'll learn

You've seen at least one receipt and want to know what happened underneath.

  • which part is yours, the run or the model
  • the order to read the concepts in
  • which pages are the data model

Each page explains one idea well enough that you can predict what gtme will do. They come in two parts, and most readers need only one.

How a run works is for anyone who runs a pipeline, whether or not they wrote it. The gate ladder says what each command spends and sends. Runs and receipts reads the table a run ends with.

The ledger is where every fact goes, and it's why the second run of a pipeline is cheaper than the first. Groups and segments is where a run's decisions become a set the next pipeline can read. Participants is how a person or an agent answers a step. A campaign is a folder is how a finished run travels to a teammate or another machine.

How the model works is for anyone writing or extending a pipeline. A pipeline is a YAML file and steps and roles follow a file from authoring to a plan. Identity keys, canonical fields, and provenance are the three pages every guide depends on. Types and traverse is how a pipeline grows past one type of record, and two adapter tiers is how it grows past the adapters gtme ships.

The pages are also numbered for a linear read, simplest first, where each page only needs the ones before it. If you haven't run anything yet, Start will get you a receipt in a few minutes, and the concepts read better with one in front of you.

  • The gate ladder — How a run works, first page; what each command spends and sends
  • Runs and receipts — How a run works; the table a run ends with, column by column
  • The ledger — The one concept the rest depend on; facts live in an append-only table and steps read from it instead of passing records
  • Groups and segments — How a run works; where one run's decisions become a set the next pipeline reads
  • Participants — How a run works; how a person or an agent answers a step and the run collects it
  • A campaign is a folder — How a run works; a frozen run travels as a folder of text
  • A pipeline is a YAML file — How the model works, first page; every key in a pipeline file
  • Steps and roles — How the model works; what each role reads, writes, and costs
  • Identity keys — How the model works; what makes a row the same record across runs and vendors
  • Canonical fields — How the model works; one name with one meaning across adapters
  • Facts have provenance — How the model works; which value wins when two rows disagree
  • Types and traverse — How the model works; a run past one type of record
  • Two adapter tiers — How the model works; where an adapter comes from and what each tier can express
  • Start — Concepts assume you've seen at least one receipt; Start is where you get one