Skip to content

ADR 0009: Artifact delivery and cache materializer

Status

Accepted.

Amended by ADR 0024. Its strict v2 wire, fixed pod commands, runtime receipt, and trust semantics govern executable init_fetch.

Context

The Airflow scheduler, worker, and Kubernetes runtime pod usually do not share one filesystem. The Airflow loader must not synchronize remote artifacts during DAG parsing.

Decision

Separate four roles:

  • Publisher writes immutable release/deployment artifacts.
  • Cache Materializer verifies a local candidate, serializes promotion through one cache-root inter-process lock, checks CAS under that lock, and switches current only as the final activation commit point.
  • Recovery Planner independently validates active state and every candidate. Recovery Apply requires an allowed actor and the active-state CAS value from the reviewed plan.
  • Airflow Loader reads only local current/airflow-index.json.

For KPO runtime, the primary production delivery mode is strict-v2 init_fetch: an init container downloads pinned artifacts by digest into an emptyDir, validates deployment-set.v2 as the runtime deployment receipt, independently validates release-set.v1 and the selected pack, applies the effective trust policy, then publishes runtime-fetch-ready.json last. The base launcher reads that immutable ready state and fetched artifacts; it never resolves current or reopens init-only ConfigMap mounts.

Consequences

  • Airflow parse remains network-free.
  • Runtime pods can fetch pinned artifacts without resolving mutable current.
  • Artifact credentials stay out of KPO arguments and generated packs.
  • deployment_id is recomputed before promotion. Environment-specific fields mirrored into airflow-index.json must equal deployment.json.
  • The pinned release-set and each DAG/pack pass confinement, size, and SHA-256 validation before control-state mutation. File opens walk from an anchored cache-root descriptor and reject intermediate symlink replacement.
  • The materializer prepares a unique relative symlink, durably replaces pointer JSON, appends promotion authorization, then atomically replaces current.
  • current targets a sealed activations/<environment>/<deployment_id> snapshot, not the mutable build candidate. The materializer copies, seals, compares the complete candidate/staging tree, and revalidates that snapshot and the pinned release tree before pointer or audit mutation. Existing activations are reused only after exact tree comparison. Unreadable subtrees fail closed, and non-regular entries are opened nonblocking before rejection.
  • Lock, pointer, and audit control files reject symlinks. Pointer temporary files are unique/exclusive and fsynced; audit append uses no-follow semantics.
  • Promotion, recovery apply, and retention apply hold the same cache-root transaction lock from fresh validation/plan through their mutation.
  • Cache-path and control identities are canonical lowercase SHA-256 strings. Every current consumer validates publisher and offset-aware promotion time in addition to environment, release, and deployment identity.
  • --expect-current-absent protects the first promotion. Later promotions use --expected-current-deployment-id. Concurrent guarded promotions therefore have one winner; stale callers receive DPONE_CURRENT_POINTER_CAS_MISMATCH.
  • A pointer-write failure leaves active current unchanged, but can leave a visible pointer when failure occurs after rename and before directory fsync. Audit or activation failure can leave prepared metadata ahead of active state and returns failed_step, state_may_have_changed, and recovery_required.
  • A failure while preparing the current symlink leaves pointer, audit, and active current unchanged, but the sealed activation or release permissions may already have changed. It therefore reports possible artifact mutation without claiming control-state recovery is required.
  • Retention validates the full delete set before mutation. A later filesystem failure reports completed deployment IDs, the failed ID, and the failed step; it never converts partial deletion into an opaque exception.
  • Retention fully validates active current before classifying or deleting any other deployment. A corrupt current fingerprint, index, release, or artifact therefore preserves all potential recovery candidates.
  • Recovery never offers an unverified candidate. It refuses healthy-state mutation and stale-plan rollback. Pointer and active projection environment, deployment, and release identities must agree before audit-only repair. Existing noncanonical current paths with no physical CAS identity block automatic recovery. Audit-only repair preserves original publisher, commit, and attestation fields.
  • Malformed pointer and directory identities are represented as null plus a structured issue/quarantine record, never as invalid digest text in public recovery or retention reports. Retention schemas prohibit null identities on destructive actions, and human output marks quarantine as NEEDS_ATTENTION instead of reporting an apparently healthy plan.
  • Historical malformed audit lines are warnings and are never silently erased.
  • CLI actor/allowlist values enforce policy consistency but do not authenticate the process. CI/workload identity and cache write permissions are the trust boundary.

Promotion State Machine

unlocked
  -> locked
  -> candidate_verified
  -> activation_snapshot_sealed
  -> cas_verified
  -> pointer_prepared
  -> audit_authorized
  -> current_activated
  -> unlocked/success

Any failure before current_activated is not a successful deployment activation. If pointer or audit metadata was already prepared, the operator must run the plan/apply workflow instead of retrying promotion blindly.

Rejected Alternatives

  • Check-then-act CAS without a lock: rejected because two promoters can both report success for different deployments.
  • Fixed temporary filenames: rejected because concurrent writers collide and a symlink can redirect writes outside the cache root.
  • Activate current before pointer/audit: rejected because a command can fail after making an unaudited deployment visible to the loader.
  • Directory-copy fallback for current: rejected because it has no atomic activation boundary. Copying into an immutable activation snapshot before the atomic symlink switch is accepted because the switch remains the single activation boundary.
  • Recovery by arbitrary deployment ID: rejected because it bypasses candidate integrity, promoter policy, and reviewed-state concurrency guards.

Compatibility

This security correction is intentionally fail-closed. Existing caches must be rematerialized when deployment identity does not match its content, when the release-set is absent, or when current is not a relative symlink. Rolling back to the former materializer removes these integrity and concurrency guarantees. The operational migration is specified in the cache sync and recovery runbook.