Live certification¶

Live certification proves connector and strategy behavior against disposable local services or real vendor systems. It is manual by design and complements the default non-live CI gate.

Contents¶

Profiles
Local live flow
Vendor live flow
Benchmark and SLO gate
Performance, state, and release evidence
Artifacts
GitHub Actions
Runbook
Developer notes

Profiles¶

Profile	Credentials	Services	Purpose
`local_live`	no external credentials	Postgres, MSSQL, ClickHouse, Kafka, Schema Registry, MinIO	Prove native clients, staging, state, matrix behavior, metrics, benchmark/SLO, and evidence chain on disposable local services.
`real_local`	no external credentials	Postgres, MSSQL, ClickHouse, Kafka, Schema Registry, MinIO	Run the same disposable stack with `DPONE_MATRIX_RUN_MODE=real_local` and produce release-grade performance, state/reconciliation, and release evidence artifacts.
`vendor_live`	yes	real managed/vendor systems	Prove provider/API behavior, quota handling, external auth, and managed-system compatibility before a release.

Use local_live during feature work, real_local before minor/major releases, and vendor_live when provider/API behavior must be proven against managed systems. The vendor_live profile is intentionally manual and requires configured CI secrets or local secret providers.

Local live flow¶

Generate the plan:

dpone ops live-certification-plan \
  --profile local_live \
  --row-count 25000 \
  --output-dir test_artifacts/live_certification/plan \
  --format json

Start local services:

docker compose -f docker/docker-compose.integration.yml up -d \
  postgres mssql clickhouse kafka schema-registry minio

Run service markers:

DPONE_RUN_INTEGRATION=1 \
uv run pytest \
  -m "integration_postgres or integration_mssql or integration_clickhouse or integration_kafka" \
  tests/integration \
  -q

Run the source -> sink matrix:

DPONE_RUN_INTEGRATION=1 \
DPONE_RUN_INTEGRATION_MATRIX=1 \
DPONE_MATRIX_RUN_MODE=mock_local \
DPONE_MATRIX_ROW_COUNT=25000 \
DPONE_MATRIX_ARTIFACT_DIR=test_artifacts/live_certification/matrix \
uv run pytest -m integration_matrix tests/integration/matrix -q

For a release-candidate local gate, switch the matrix mode to real_local:

DPONE_RUN_INTEGRATION=1 \
DPONE_RUN_INTEGRATION_MATRIX=1 \
DPONE_MATRIX_RUN_MODE=real_local \
DPONE_MATRIX_ROW_COUNT=25000 \
DPONE_MATRIX_ARTIFACT_DIR=test_artifacts/live_certification/matrix \
uv run pytest -m integration_matrix tests/integration/matrix -q

Stop services after artifacts are collected:

docker compose -f docker/docker-compose.integration.yml down -v

Vendor live flow¶

Generate a vendor plan:

dpone ops live-certification-plan \
  --profile vendor_live \
  --include-vendor-live \
  --row-count 10000 \
  --output-dir test_artifacts/live_certification_vendor/plan \
  --format json

Run vendor tests only when secrets and cost controls are configured:

DPONE_RUN_INTEGRATION_LIVE=1 \
uv run pytest -m integration_live tests/integration -q

Never put vendor credentials in manifests, workflow logs, or certification artifacts.

Benchmark and SLO gate¶

Use benchmark-slo-gate to combine performance regression checks and operational SLO objectives:

dpone ops benchmark-slo-gate \
  --metrics-json '{"throughput_rows_per_second":120000,"duration_seconds":40,"freshness_lag_seconds":120,"failure_rate":0}' \
  --baseline-json '{"throughput_rows_per_second":{"value":100000,"direction":"higher"},"duration_seconds":{"value":60,"direction":"lower"}}' \
  --objectives-json '{"throughput_rows_per_second":{"min":100000},"freshness_lag_seconds":{"max":300},"failure_rate":{"max":0}}' \
  --output-dir test_artifacts/live_certification/benchmark-slo \
  --format json

Generated files:

File	Purpose
`benchmark_slo_gate.json`	Machine-readable combined benchmark and SLO decision.
`benchmark_slo_gate.md`	Human-readable report and runbook.
`benchmark/benchmark_baseline.json`	Underlying performance baseline report.
`benchmark/benchmark_baseline.md`	Underlying performance runbook.

The gate is red when either benchmark regression or SLO evaluation is red.

Performance, state, and release evidence¶

real_local and release-candidate runs must produce three additional artifacts:

dpone ops performance-certification \
  --profile real_local \
  --row-count 25000 \
  --metrics-json '{"throughput_rows_per_second":1000,"duration_seconds":60,"memory_peak_mb":1024,"failure_rate":0}' \
  --minimum-json '{"throughput_rows_per_second":500,"failure_rate":0}' \
  --maximum-json '{"duration_seconds":120,"memory_peak_mb":2048}' \
  --output-dir test_artifacts/live_certification/performance-certification \
  --format json

dpone ops live-state-reconciliation \
  --profile real_local \
  --artifact state=test_artifacts/live_certification/state_evidence.json \
  --artifact reconciliation=test_artifacts/live_certification/reconciliation_evidence.json \
  --require state \
  --require reconciliation \
  --output-dir test_artifacts/live_certification/live-state-reconciliation \
  --format json

dpone ops pre-release-checklist \
  --release v0.7.1 \
  --release-type minor \
  --check cli_commands=true \
  --check run_cli=true \
  --check run_python_api=true \
  --check hierarchical_lineage=true \
  --check source_sink_matrix=true \
  --check contracts_guardrails=true \
  --check documentation=true \
  --check package=true \
  --output-dir test_artifacts/live_certification/pre-release \
  --format json

dpone ops release-evidence-pack \
  --release v0.7.1 \
  --profile real_local \
  --artifact service_markers=test_artifacts/live_certification/service_markers.json \
  --artifact certification_pack=test_artifacts/live_certification/certification-pack/connector_certification_pack.json \
  --artifact performance_certification=test_artifacts/live_certification/performance-certification/performance_certification.json \
  --artifact live_state_reconciliation=test_artifacts/live_certification/live-state-reconciliation/live_state_reconciliation.json \
  --artifact evidence_chain=test_artifacts/live_certification/evidence-chain/evidence_chain_index.json \
  --artifact pre_release_checklist=test_artifacts/live_certification/pre-release/pre_release_checklist.json \
  --require service_markers \
  --require certification_pack \
  --require performance_certification \
  --require live_state_reconciliation \
  --require evidence_chain \
  --require pre_release_checklist \
  --output-dir test_artifacts/live_certification/release-evidence \
  --format json

Minor and major releases must not be tagged until release_evidence_pack.json is green. Patch releases may use the same gate when runtime, CLI, schema, strategy, connector, or documentation behavior changes.

Artifacts¶

Recommended artifact tree:

test_artifacts/live_certification/
  plan/live_certification_plan.json
  matrix/certification_report.json
  observability/runtime_metrics.json
  benchmark-slo/benchmark_slo_gate.json
  performance-certification/performance_certification.json
  live-state-reconciliation/live_state_reconciliation.json
  certification-pack/connector_certification_pack.json
  artifact-index/artifact_index.json
  evidence-chain/evidence_chain_index.json
  release-evidence/release_evidence_pack.json

Attach benchmark_slo_gate.json, connector_certification_pack.json, and evidence_chain_index.json to release or industrial-readiness evidence. For minor/major releases, attach release_evidence_pack.json as the top-level go/no-go decision.

GitHub Actions¶

The repository includes .github/workflows/live-certification.yml.

It is manual-only and supports:

local_live: starts Docker services from docker/docker-compose.integration.yml, runs service markers, matrix tests, observability export, benchmark-slo-gate, certification pack, artifact index, and evidence chain.
real_local: uses the same Docker stack, sets DPONE_MATRIX_RUN_MODE=real_local, and additionally builds performance-certification, live-state-reconciliation, and release-evidence-pack artifacts.
vendor_live: runs real provider tests only when run_vendor_live=true and secrets are configured.

Runbook¶

Failure: local services fail to start.

Inspect Docker service logs.
Verify SQL Server memory and password complexity.
Verify Kafka is healthy before Schema Registry.
Re-run local_live after fixing runner capacity.

Failure: matrix case is red.

Open <case_id>__behavior.json.
Re-run with DPONE_MATRIX_CASE_ID=<case_id>.
Check the source -> sink guide for that pair.
Do not update expected behavior until the strategy contract is reviewed.

Failure: benchmark-slo-gate is red.

Compare current metrics with committed baselines.
Check native fast path, partitioning, batch size, finalizer policy, and target capacity.
Do not loosen SLOs to hide a regression.
Update baselines only after an intentional reviewed performance change.

Failure: evidence chain is red.

Rebuild artifact index.
Verify required artifacts exist and have stable checksums.
Re-run dpone ops evidence-chain.
Treat checksum drift as an audit event.

Failure: release-evidence-pack is red.

Open release_evidence_pack.json and inspect blockers.
If an artifact is missing, re-run the originating gate instead of editing the pack.
If an artifact is red, fix the source failure first.
Do not create a minor or major release tag until the pack is green.

Developer notes¶

Extension	Module
Live certification plan model	`dpone.ops.live_certification`
Combined benchmark/SLO gate	`dpone.ops.benchmark_slo_gate`
Performance certification	`dpone.ops.performance_certification`
State/reconciliation certification	`dpone.ops.live_state_reconciliation`
Release evidence pack	`dpone.ops.release_evidence_pack`
Matrix registry	`dpone.integration_matrix`
Manual workflow	`.github/workflows/live-certification.yml`

Keep this layer dependency-light. It should orchestrate evidence and commands, not import heavy database or Kafka clients.