Live certification

Live certification proves connector and strategy behavior against disposable local services or real vendor systems. It is manual by design and complements the default non-live CI gate.

Contents

• Profiles
• Local live flow
• Vendor live flow
• Benchmark and SLO gate
• Performance, state, and release evidence
• Artifacts
• GitHub Actions
• Runbook
• Developer notes

Profiles

Profile	Credentials	Services	Purpose
local_live	no external credentials	Postgres, MSSQL, ClickHouse, Kafka, Schema Registry, MinIO	Prove native clients, staging, state, matrix behavior, metrics, benchmark/SLO, and evidence chain on disposable local services.
real_local	no external credentials	Postgres, MSSQL, ClickHouse, Kafka, Schema Registry, MinIO	Run the same disposable stack with DPONE_MATRIX_RUN_MODE=real_local and produce release-grade performance, state/reconciliation, and release evidence artifacts.
type_matrix_certification	no external credentials	Postgres, MSSQL, ClickHouse	Validate critical-route type decisions, physical DDL, temporal fidelity, schema-evolution reruns and local live fixtures for MSSQL -> ClickHouse and Postgres -> MSSQL.
native_transfer	no external credentials	Postgres, MSSQL, ClickHouse, Kafka, Schema Registry, MinIO	Validate critical native transfer routes and build native evidence -> strategy certification bundle -> release evidence pack.
vendor_live	yes	real managed/vendor systems	Prove provider/API behavior, quota handling, external auth, and managed-system compatibility before a release.

Use local_live during feature work, type_matrix_certification for critical type/DDL/temporal-fidelity changes, native_transfer for critical native route certification, real_local before minor/major releases, and vendor_live when provider/API behavior must be proven against managed systems. The vendor_live profile is intentionally manual and requires configured CI secrets or local secret providers.

Local live flow

Generate the plan:

dpone ops live-certification-plan \
  --profile local_live \
  --row-count 25000 \
  --output-dir test_artifacts/live_certification/plan \
  --format json

Start local services:

docker compose -f docker/docker-compose.integration.yml up -d \
  postgres mssql clickhouse kafka schema-registry minio

Run service markers:

DPONE_RUN_INTEGRATION=1 \
uv run pytest \
  -m "integration_postgres or integration_mssql or integration_clickhouse or integration_kafka" \
  tests/integration \
  -q

Run the source -> sink matrix:

DPONE_RUN_INTEGRATION=1 \
DPONE_RUN_INTEGRATION_MATRIX=1 \
DPONE_MATRIX_RUN_MODE=mock_local \
DPONE_MATRIX_ROW_COUNT=25000 \
DPONE_MATRIX_ARTIFACT_DIR=test_artifacts/live_certification/matrix \
uv run pytest -m integration_matrix tests/integration/matrix -q

For a release-candidate local gate, switch the matrix mode to real_local:

DPONE_RUN_INTEGRATION=1 \
DPONE_RUN_INTEGRATION_MATRIX=1 \
DPONE_MATRIX_RUN_MODE=real_local \
DPONE_MATRIX_ROW_COUNT=25000 \
DPONE_MATRIX_ARTIFACT_DIR=test_artifacts/live_certification/matrix \
uv run pytest -m integration_matrix tests/integration/matrix -q

Stop services after artifacts are collected:

docker compose -f docker/docker-compose.integration.yml down -v

Vendor live flow

Generate a vendor plan:

dpone ops live-certification-plan \
  --profile vendor_live \
  --include-vendor-live \
  --row-count 10000 \
  --output-dir test_artifacts/live_certification_vendor/plan \
  --format json

Run vendor tests only when secrets and cost controls are configured:

uv run dpone ops managed-credentials-readiness \
  --profile vendor_live \
  --required-env APPSFLYER_API_TOKEN \
  --required-env GOOGLE_ADS_DEVELOPER_TOKEN \
  --output-dir test_artifacts/live_certification_vendor/managed-credentials \
  --format json

DPONE_RUN_INTEGRATION_LIVE=1 \
uv run pytest -m integration_live tests/integration -q

managed-credentials-readiness records only credential variable names and presence/missing status. It never reads, prints, hashes, or stores credential values. The GitHub vendor_live workflow runs this gate before the backfill matrix, so missing managed credentials fail fast before any external API or database test spends quota.

Never put vendor credentials in manifests, workflow logs, or certification artifacts. If a managed system needs a stronger probe than environment presence (for example, API whoami or bucket sentinel access), implement that as a connector capability probe with redacted evidence rather than extending this generic readiness command.

Benchmark and SLO gate

Use benchmark-slo-gate to combine performance regression checks and operational SLO objectives:

dpone ops benchmark-slo-gate \
  --metrics-json '{"throughput_rows_per_second":120000,"duration_seconds":40,"freshness_lag_seconds":120,"failure_rate":0}' \
  --baseline-json '{"throughput_rows_per_second":{"value":100000,"direction":"higher"},"duration_seconds":{"value":60,"direction":"lower"}}' \
  --objectives-json '{"throughput_rows_per_second":{"min":100000},"freshness_lag_seconds":{"max":300},"failure_rate":{"max":0}}' \
  --output-dir test_artifacts/live_certification/benchmark-slo \
  --format json

Generated files:

File	Purpose
benchmark_slo_gate.json	Machine-readable combined benchmark and SLO decision.
benchmark_slo_gate.md	Human-readable report and runbook.
benchmark/benchmark_baseline.json	Underlying performance baseline report.
benchmark/benchmark_baseline.md	Underlying performance runbook.

The gate is red when either benchmark regression or SLO evaluation is red.

Performance, state, and release evidence

real_local and release-candidate runs must produce three additional artifacts:

dpone ops performance-certification \
  --profile real_local \
  --row-count 25000 \
  --metrics-json '{"throughput_rows_per_second":1000,"duration_seconds":60,"memory_peak_mb":1024,"failure_rate":0}' \
  --minimum-json '{"throughput_rows_per_second":500,"failure_rate":0}' \
  --maximum-json '{"duration_seconds":120,"memory_peak_mb":2048}' \
  --output-dir test_artifacts/live_certification/performance-certification \
  --format json

dpone ops live-state-reconciliation \
  --profile real_local \
  --artifact state=test_artifacts/live_certification/state_evidence.json \
  --artifact reconciliation=test_artifacts/live_certification/reconciliation_evidence.json \
  --require state \
  --require reconciliation \
  --output-dir test_artifacts/live_certification/live-state-reconciliation \
  --format json

dpone ops pre-release-checklist \
  --release vX.Y.Z \
  --release-type minor \
  --check cli_help_surface=true \
  --check cli_output_contracts=true \
  --check run_cli_manifest=true \
  --check run_python_api_manifest=true \
  --check nested_hierarchical_identity=true \
  --check nested_parent_child_integrity=true \
  --check source_sink_strategy_matrix=true \
  --check source_sink_artifacts=true \
  --check docker_live_routes=true \
  --check contracts_guardrails=true \
  --check documentation_yaml_examples=true \
  --check documentation_links=true \
  --check documentation_mkdocs=true \
  --check ci_cd_quality=true \
  --check package=true \
  --output-dir test_artifacts/live_certification/pre-release \
  --format json

dpone ops release-evidence-pack \
  --release vX.Y.Z \
  --profile real_local \
  --artifact service_markers=test_artifacts/live_certification/service_markers.json \
  --artifact certification_pack=test_artifacts/live_certification/certification-pack/connector_certification_pack.json \
  --artifact performance_certification=test_artifacts/live_certification/performance-certification/performance_certification.json \
  --artifact live_state_reconciliation=test_artifacts/live_certification/live-state-reconciliation/live_state_reconciliation.json \
  --artifact evidence_chain=test_artifacts/live_certification/evidence-chain/evidence_chain_index.json \
  --artifact pre_release_checklist=test_artifacts/live_certification/pre-release/pre_release_checklist.json \
  --require service_markers \
  --require certification_pack \
  --require performance_certification \
  --require live_state_reconciliation \
  --require evidence_chain \
  --require pre_release_checklist \
  --output-dir test_artifacts/live_certification/release-evidence \
  --format json

Minor and major releases must not be tagged until release_evidence_pack.json is green. Patch releases may use the same gate when runtime, CLI, schema, strategy, connector, or documentation behavior changes.

Artifacts

Recommended artifact tree:

test_artifacts/live_certification/
  plan/live_certification_plan.json
  matrix/certification_report.json
  observability/runtime_metrics.json
  benchmark-slo/benchmark_slo_gate.json
  performance-certification/performance_certification.json
  live-state-reconciliation/live_state_reconciliation.json
  certification-pack/connector_certification_pack.json
  artifact-index/artifact_index.json
  evidence-chain/evidence_chain_index.json
  release-evidence/release_evidence_pack.json

Attach benchmark_slo_gate.json, connector_certification_pack.json, and evidence_chain_index.json to release or industrial-readiness evidence. For minor/major releases, attach release_evidence_pack.json as the top-level go/no-go decision.

GitHub Actions

The repository includes .github/workflows/live-certification.yml.

It is manual-only and supports:

• local_live: starts Docker services from docker/docker-compose.integration.yml, runs service markers, matrix tests, observability export, benchmark-slo-gate, certification pack, artifact index, and evidence chain.
• real_local: uses the same Docker stack, sets DPONE_MATRIX_RUN_MODE=real_local, and additionally builds performance-certification, live-state-reconciliation, and release-evidence-pack artifacts.
• vendor_live: runs real provider tests only when run_vendor_live=true and secrets are configured.
• route live certification: use dpone ops route-live-certification after Docker-live or vendor-live route artifacts exist, then feed route_live_evidence_bundle to dpone ops route-release-gate.
• route release candidate orchestrator: use dpone ops route-rc-orchestrator when the release candidate should produce route certification, route live certification, route release gate, and release evidence pack outputs in one receipt.
• route release candidate executor: use dpone ops route-rc-execute to dry-run route_rc_orchestration.json receipts by default, or set execute_route_rc_commands=true in the manual workflow to run the same commands against the disposable Docker-live services before teardown.

Runbook

Failure: local services fail to start.

1. Inspect Docker service logs.
2. Verify SQL Server memory and password complexity.
3. Verify Kafka is healthy before Schema Registry.
4. Re-run local_live after fixing runner capacity.

Failure: matrix case is red.

1. Open <case_id>__behavior.json.
2. Re-run with DPONE_MATRIX_CASE_ID=<case_id>.
3. Check the source -> sink guide for that pair.
4. Do not update expected behavior until the strategy contract is reviewed.

Failure: benchmark-slo-gate is red.

1. Compare current metrics with committed baselines.
2. Check native fast path, partitioning, batch size, finalizer policy, and target capacity.
3. Do not loosen SLOs to hide a regression.
4. Update baselines only after an intentional reviewed performance change.

Failure: evidence chain is red.

1. Rebuild artifact index.
2. Verify required artifacts exist and have stable checksums.
3. Re-run dpone ops evidence-chain.
4. Treat checksum drift as an audit event.

Failure: release-evidence-pack is red.

1. Open release_evidence_pack.json and inspect blockers.
2. If an artifact is missing, re-run the originating gate instead of editing the pack.
3. If an artifact is red, fix the source failure first.
4. Do not create a minor or major release tag until the pack is green.

Developer notes

Extension	Module
Live certification plan model	dpone.ops.live_certification
Combined benchmark/SLO gate	dpone.ops.benchmark_slo_gate
Performance certification	dpone.ops.performance_certification
State/reconciliation certification	dpone.ops.live_state_reconciliation
Release evidence pack	dpone.ops.release_evidence_pack
Matrix registry	dpone.integration_matrix
Manual workflow	.github/workflows/live-certification.yml

Keep this layer dependency-light. It should orchestrate evidence and commands, not import heavy database or Kafka clients.

Native transfer profile route coverage

The native_transfer profile certifies both critical native routes as one release-gate chain:

• Postgres -> MSSQL: builds postgres-mssql/evidence/.../evidence_index.json.
• MSSQL -> ClickHouse: builds mssql-clickhouse/evidence/.../evidence_index.json.

Both evidence indexes are passed to dpone strategy certification-bundle with repeated --native-transfer-evidence flags. The resulting strategy_certification_bundle.json is then required by dpone ops release-evidence-pack --profile native_transfer.