Online schema evolution¶

dpone online schema evolution adds DDL governance on top of the existing safe schema comparator. The comparator detects drift; the online planner decides whether each DDL action is metadata-only, low-lock, blocking, breaking, or unsupported.

Default policy¶

sink:
  options:
    schema_evolution:
      enabled: true
      mode: widening
      tables: evolve
      columns: evolve
      data_type: widen
      ddl_mode: online
      on_schema_change: apply
      on_type_change: fail
      new_column_prefix: "__dpone__nc__"

Only non-breaking, online-safe changes are auto-applied. Blocking changes are deferred in ddl_mode: online unless allow_blocking_online: true is explicitly set.

Contract modes¶

Entity	Modes	Default	Notes
`tables`	`evolve`, `freeze`, `ignore`	`evolve`	Table-level creation/drop remains conservative.
`columns`	`evolve`, `freeze`, `ignore`, `quarantine`	`evolve`	New columns can be applied, frozen, ignored, or routed to quarantine evidence.
`data_type`	`widen`, `variant_column`, `freeze`, `quarantine`	`widen`	`variant_column` uses `__dpone__nc__<column>` for incompatible values.
`ddl_mode`	`online`, `safe_window`, `plan_only`, `manual_approval`	`online`	Controls whether planned DDL can execute automatically.

DDL modes¶

Mode	Behavior
`online`	Apply metadata-only/low-lock DDL; reject blocking/breaking DDL before staging.
`safe_window`	Allow safe DDL during an operator-approved maintenance window.
`plan_only`	Produce the plan and fail before load if DDL is needed.
`manual_approval`	Defer DDL and require approval before applying.

Example with lock budget and ledger¶

sink:
  options:
    schema_evolution:
      ddl_mode: online
      lock_timeout_seconds: 5
      statement_timeout_seconds: 60
      ledger_path: .dpone/schema_changes

Postgres DDL is decorated with session-level timeout statements where supported:

SET lock_timeout = '5s';
SET statement_timeout = '60s';
ALTER TABLE "landing"."orders" ADD COLUMN "status" text;

Risk classes¶

Risk	Auto online?	Examples
`metadata_only`	yes	Nullable `ADD COLUMN`, generated compatibility column.
`low_lock`	yes	BigQuery schema API widening.
`blocking`	no by default	Most DB type widening, non-null add without default-safe path.
`breaking`	no	Drop, rename, narrowing, reserved framework namespace violation.
`unsupported`	no	Kafka table DDL; use Schema Registry compatibility instead.

Expand-contract workflow¶

Breaking or blocking changes should be handled as expand-contract:

flowchart LR
    Detect["Detect drift"] --> Expand["Expand compatible schema"]
    Expand --> Route["Route new values safely"]
    Route --> Backfill["Optional chunked backfill"]
    Backfill --> Validate["Validate downstream consumers"]
    Validate --> Contract["Manual contract cleanup"]

For incompatible type changes, prefer:

sink:
  options:
    schema_evolution:
      data_type: variant_column
      on_type_change: new_column

This creates __dpone__nc__<column> and preserves the original target column.

CLI planning¶

dpone schema plan \
  --source source-columns.json \
  --target target-columns.json \
  --table landing.orders \
  --dialect postgres \
  --ddl-mode online \
  --lock-timeout-seconds 5 \
  --format json

The JSON output contains online_schema_evolution with risk levels, decisions, blockers, DDL, and expand-contract guidance.

Runbook¶

Symptom	Action
`schema_evolution.blocking:*`	Use `safe_window`, plan an expand-contract migration, or explicitly accept blocking DDL.
`schema_evolution.breaking:*`	Do not auto-apply. Create an expand-contract change request.
DDL permission denied	Grant least-privilege `ALTER`/schema update permission or use `plan_only`.
Lock timeout	Re-run during a safer window or use shadow/expand-contract migration.
Type conflict	Use `data_type: variant_column` and migrate consumers to `__dpone__nc__*`.
Kafka schema conflict	Use Schema Registry compatibility checks; no table DDL is emitted.

Evidence¶

When ledger_path is configured, runtime writes schema change ledger artifacts with:

schema_change_id
run_id
table and dialect
risk level and decision
DDL statements
blockers and expand-contract guidance

Feed this artifact into dpone ops industrial-readiness as the schema_evolution evidence domain.

Runtime governed DDL execution¶

Runtime uses sink-specific online DDL adapters before staging load:

Sink	Adapter	Behavior
Postgres	`PostgresOnlineDdlAdapter`	Applies decorated SQL through `execute_query`, including `SET lock_timeout` and `SET statement_timeout`.
MSSQL	`MssqlOnlineDdlAdapter`	Applies `SET LOCK_TIMEOUT` and safe `ALTER TABLE` actions.
ClickHouse	`ClickHouseOnlineDdlAdapter`	Applies safe `ADD COLUMN`; unsafe modifications remain blocked by governance.
BigQuery	`BigQuerySchemaUpdateAdapter`	Uses the existing connector query/API facade for schema updates.
Kafka	`KafkaSchemaRegistryCompatibilityAdapter`	Performs Schema Registry compatibility checks instead of table DDL.

If a sink exposes apply_governed_schema_plan, runtime delegates to it. Otherwise, connectors exposing execute_query receive only governed actions with decision apply. Test doubles and legacy sinks can still use apply_schema_plan as a compatibility fallback.

Approval workflow¶

Use manual_approval when DDL should be reviewed before execution:

sink:
  options:
    schema_evolution:
      ddl_mode: manual_approval
      ledger_path: .dpone/schema_changes

Approve a ledger artifact:

dpone schema approve \
  --ledger .dpone/schema_changes/schema_change_run01_abcd.json \
  --approver data-architect \
  --output-dir .dpone/schema_approvals \
  --format json

Executable expand-contract plan¶

For incompatible changes, generate a three-phase migration artifact:

dpone schema expand-contract \
  --source source-columns.json \
  --target target-columns.json \
  --table landing.orders \
  --dialect postgres \
  --output-dir .dpone/schema_expand_contract \
  --format json

The artifact contains expand, backfill, and contract phases and uses __dpone__nc__* for variant-column compatibility.

Schema history and notifications¶

SchemaHistoryRegistry records versioned schema snapshots and diffs for each table. SchemaNotificationService turns ledger artifacts into JSON and Markdown notifications that can be attached to run reports, release evidence, or future Slack/webhook integrations.