Skip to content

CI/CD workflow reference

This page documents every public GitHub Actions workflow in dpone: trigger, intent, local reproduction command, artifacts, and first troubleshooting pointer.

Workflow overview

flowchart TD
    Push["push to master"] --> CI["ci.yml quality matrix"]
    Push --> Pages["pages.yml docs deploy"]
    Push --> Secret["secret-scan.yml"]
    Push --> CodeQL["codeql.yml"]
    Push --> Scorecard["scorecard.yml"]
    PR["pull request to master"] --> CI
    PR --> PagesBuild["pages.yml build only"]
    PR --> Secret
    PR --> CodeQL
    Tag["vX.Y.Z tag"] --> Release["release.yml"]
    Manual["workflow_dispatch"] --> Matrix["integration-matrix.yml"]
    Manual --> Replay["replay-integration.yml"]
    Manual --> Cert["connector-certification.yml"]
    Manual --> Obs["observability-maturity.yml"]
    Schedule["schedule"] --> Cert
    Schedule --> Obs
    Schedule --> CodeQL
    Schedule --> Secret
    Schedule --> Scorecard

CI quality matrix

Field Value
Workflow .github/workflows/ci.yml
Name CI
Triggers push to master, pull request to master, manual dispatch
Python versions 3.11, 3.12
Required purpose Default code quality gate

Steps:

  1. Checkout.
  2. Install uv.
  3. Set up matrix Python.
  4. uv sync --all-extras.
  5. uv run ruff check ..
  6. uv run ruff format --check ..
  7. uv run mypy --config-file mypy.ini.
  8. uv run pytest -m "not integration_live" --cov=src/dpone --cov-report=xml.
  9. uv build.
  10. Upload coverage.xml and dist/.

Local reproduction:

uv sync --all-extras
uv run ruff check .
uv run ruff format --check .
uv run mypy --config-file mypy.ini
uv run pytest -m "not integration_live" --cov=src/dpone --cov-report=xml
uv build

Artifacts:

  • coverage.xml
  • dist/dpone-*.whl
  • dist/dpone-*.tar.gz

Runbook: CI quality failures.

PostgreSQL XMin integration job

Field Value
Workflow .github/workflows/ci.yml
Job postgres-xmin
Trigger same as CI
Service postgres:16-alpine
Purpose Keep the explicit Postgres XMin strategy tested in regular CI

Local reproduction with your own Postgres instance:

DPONE_RUN_INTEGRATION=1 \
DPONE_IT_PG_HOST=127.0.0.1 \
DPONE_IT_PG_PORT=5432 \
DPONE_IT_PG_DATABASE=dpone_it \
DPONE_IT_PG_USER=dpone \
DPONE_IT_PG_PASSWORD=dpone \
uv run pytest -m integration_postgres_xmin tests/integration/postgres -q

Runbook: PostgreSQL XMin integration failures.

Documentation and GitHub Pages

Field Value
Workflow .github/workflows/pages.yml
Name docs
Triggers docs/mkdocs/workflow changes on push or PR, manual dispatch
Deploy condition push to master, not pull request
Purpose Strict docs build and GitHub Pages deploy

Local reproduction:

python -m pip install -r docs/requirements.txt
mkdocs build --strict

The workflow deploys to https://paulkov.github.io/dpone/ from the generated site/ artifact.

Runbook: Docs and Pages failures.

Release publishing

Field Value
Workflow .github/workflows/release.yml
Name Release
Triggers vX.Y.Z tags and manual dispatch
Environment pypi
Preferred publish mode PyPI Trusted Publishing
Fallback publish mode PYPI_API_TOKEN secret when explicitly requested

Local reproduction before tagging:

uv sync --all-extras
uv build
uv tool run twine check dist/*

Runbook: Release and PyPI failures.

Secret Scan

Field Value
Workflow .github/workflows/secret-scan.yml
Tool TruffleHog verified secrets
Triggers push to master, pull request to master, weekly schedule
Purpose Prevent verified secrets from entering public history

Runbook: Secret scan failures.

CodeQL

Field Value
Workflow .github/workflows/codeql.yml
Tool GitHub CodeQL Python analysis
Triggers push, pull request, weekly schedule
Purpose Static security analysis

Runbook: CodeQL failures.

OSSF Scorecard

Field Value
Workflow .github/workflows/scorecard.yml
Tool OSSF Scorecard
Triggers push to master, branch protection rule changes, weekly schedule
Purpose Supply-chain security posture

Runbook: OSSF Scorecard failures.

Source/Sink integration matrix

Field Value
Workflow .github/workflows/integration-matrix.yml
Trigger manual dispatch
Modes mock_contract, mock_local, vendor_live
Purpose Source -> sink -> strategy certification artifacts

Typical manual run:

run_mode=mock_local
source_filter=*
sink_filter=*
strategy_filter=*
case_id_filter=*

Local reproduction:

DPONE_RUN_INTEGRATION=1 \
DPONE_RUN_INTEGRATION_MATRIX=1 \
DPONE_MATRIX_RUN_MODE=mock_contract \
DPONE_MATRIX_ARTIFACT_DIR=test_artifacts/integration_matrix/mock_contract_latest \
uv run pytest -m integration_matrix tests/integration/matrix -q

Runbook: Integration matrix failures.

Live certification

Field Value
Workflow .github/workflows/live-certification.yml
Trigger manual dispatch
Modes local_live, real_local, vendor_live
Purpose Service-backed connector, source/sink matrix, native transfer and release evidence

The local live workflow starts disposable Postgres, MSSQL, ClickHouse, Kafka, Schema Registry and MinIO services. The short default run keeps CI cost bounded by using row_count; release reviewers can additionally enable the native benchmark suite.

Typical full native transfer benchmark dispatch:

profile=real_local
row_count=100000
run_native_benchmark_suite=true
native_benchmark_rows=10000,1000000,10000000
native_benchmark_partitions=4,8
run_vendor_live=false

Artifacts:

  • test_artifacts/live_certification/benchmarks/postgres_mssql_native_fast_path.json
  • test_artifacts/live_certification/benchmarks/native_benchmark_suite/summary.json
  • test_artifacts/live_certification/benchmarks/postgres_mssql_native_benchmark_summary.md
  • test_artifacts/live_certification/matrix/certification_report.json
  • test_artifacts/live_certification/release-evidence/
  • test_artifacts/live_certification/evidence-chain/

Runbook: Live certification failures.

Strategy certification bundle:

  • The workflow first runs dpone ops integration-matrix-report to aggregate per-case *.json and *__behavior.json artifacts into certification_report.json.
  • The workflow runs dpone strategy certification-bundle with --matrix-artifact after the matrix gate, even when the matrix is red.
  • Artifact name: strategy-certification-matrix.
  • Files: strategy_certification_bundle.json, strategy_certification_bundle.md.
  • Use this bundle first during release or connector certification review because it contains checksums, docs links, and blockers.

Certification suite:

  • The workflow runs dpone ops certification-suite after the strategy bundle is generated.
  • Artifact name: source-sink-certification-suite.
  • Files: certification_suite.json, certification_suite.md, certification_suite_index.json.
  • The suite consumes test_artifacts/integration_matrix/certification_report.json and test_artifacts/strategy_certification/matrix/strategy_certification_bundle.json.

Tamper-evident evidence chain:

  • This is the tamper-evident evidence chain for the source/sink certification run.
  • The workflow runs dpone ops artifact-index over matrix, strategy bundle, and certification suite roots.
  • The workflow then runs dpone ops evidence-chain against the generated artifact_index.json.
  • The workflow finally runs dpone ops evidence-chain-verify; an empty or broken chain is a failed certification gate.
  • Artifact name: source-sink-evidence-chain.
  • Files include artifact_index.json, artifact_index.md, <release>__evidence_chain.json, <release>__evidence_chain.md, and evidence_chain_index.json.

Replay integration gate

Field Value
Workflow .github/workflows/replay-integration.yml
Trigger manual dispatch
Modes injected replay adapters, optional service-backed local Docker adapters
Purpose Validate replay/resync/resume execution paths and publish replay evidence

Typical manual run:

run_replay_gate=true
run_service_backed_gate=true

Local reproduction:

DPONE_RUN_INTEGRATION_REPLAY=1 \
uv run pytest -m integration_replay tests/integration/replay -q

docker compose -f docker/docker-compose.integration.yml up -d postgres kafka schema-registry clickhouse mssql
DPONE_RUN_INTEGRATION_REPLAY_SERVICES=1 \
uv run pytest -m integration_replay_services tests/integration/replay -q
docker compose -f docker/docker-compose.integration.yml down -v

Artifacts:

  • test_artifacts/replay_integration/junit.xml.
  • test_artifacts/replay_integration/service_junit.xml when service-backed mode is enabled.
  • strategy_certification_bundle.json.
  • strategy_certification_bundle.md.

Strategy certification bundle:

  • The workflow runs dpone strategy certification-bundle with --replay-evidence after cleanup, even when replay fails.
  • Artifact name: strategy-certification-replay.
  • Docs link embedded in the bundle: docs/testing/replay-integration.md.

Tamper-evident evidence chain:

  • This is the tamper-evident evidence chain for replay execution evidence.
  • The workflow runs dpone ops artifact-index over replay integration and strategy bundle roots.
  • The workflow then runs dpone ops evidence-chain against the generated artifact_index.json.
  • The workflow finally runs dpone ops evidence-chain-verify; an empty or broken chain is a failed replay gate.
  • Artifact name: replay-evidence-chain.
  • Files include artifact_index.json, artifact_index.md, <release>__evidence_chain.json, <release>__evidence_chain.md, and evidence_chain_index.json.

Runbook: Replay integration.

Connector certification

Field Value
Workflow .github/workflows/connector-certification.yml
Triggers daily schedule, manual dispatch
Jobs offline certification, local-live certification, vendor-live certification
Purpose Connector capability evidence and long-running confidence

Artifacts:

  • test_artifacts/connectors/matrix.json
  • test_artifacts/connectors/matrix.md
  • test_artifacts/connectors/exit-codes.json
  • strategy_certification_bundle.json
  • strategy_certification_bundle.md
  • certification_suite.json
  • certification_suite.md
  • certification_suite_index.json

Strategy certification bundle:

  • The offline job runs dpone strategy certification-bundle with --connector-artifact after generating the connector matrix.
  • Artifact name: strategy-certification-connectors.
  • Docs links embedded in the bundle: docs/connector-certification.md, docs/certification-suite.md.

Certification suite:

  • The offline job runs dpone ops certification-suite after the strategy bundle is generated.
  • Artifact name: connector-certification-suite.
  • The suite consumes test_artifacts/connectors/matrix.json and test_artifacts/strategy_certification/connectors/strategy_certification_bundle.json.

Tamper-evident evidence chain:

  • This is the tamper-evident evidence chain for the connector certification run.
  • The offline job runs dpone ops artifact-index over connector, strategy bundle, and certification suite roots.
  • The offline job then runs dpone ops evidence-chain against the generated artifact_index.json.
  • The offline job finally runs dpone ops evidence-chain-verify; an empty or broken chain is a failed certification gate.
  • Artifact name: connector-evidence-chain.
  • Files include artifact_index.json, artifact_index.md, <release>__evidence_chain.json, <release>__evidence_chain.md, and evidence_chain_index.json.

Runbook: Connector certification failures.

Certification release summary workflow

Workflow: .github/workflows/certification-release-summary.yml.

This manual workflow is the final go/no-go aggregation layer. It accepts the run IDs from replay, source -> sink matrix, and connector certification workflows, downloads their published artifacts with gh run download, and runs dpone ops release-summary.

Required source artifacts:

  • replay-evidence-chain
  • source-sink-certification-suite
  • source-sink-evidence-chain
  • connector-certification-suite
  • connector-evidence-chain

Published artifact:

  • certification-release-summary

Runbook when it fails:

  1. Open release_summary.md and identify the failing blocker.
  2. Re-run only the red upstream workflow first.
  3. Verify the upstream evidence chain with dpone ops evidence-chain-verify.
  4. Re-run .github/workflows/certification-release-summary.yml with the new upstream run ID.
  5. Do not publish a release while release-summary-report is red.

Orchestration maturity workflow

Workflow: .github/workflows/orchestration-maturity.yml.

This manual and weekly scheduled workflow validates the production orchestration profile without vendor credentials. It runs tests/test_orchestration.py, verifies documentation links with dpone docs check-docs, builds GitHub Pages with mkdocs build --strict, and uploads orchestration-maturity-report.

The gate covers:

  • local concurrency locks;
  • durable job state transitions;
  • fail-closed resume policy;
  • explicit resume and restart operator policies;
  • cron, Airflow, Dagster, and Kubernetes handoff snippets;
  • CLI and documentation contracts.

Observability maturity workflow

Workflow: .github/workflows/observability-maturity.yml.

This manual and weekly scheduled workflow validates the runtime telemetry contract without external monitoring services. It runs tests/test_observability.py, exports Prometheus and OpenTelemetry-compatible metrics with dpone observability metrics-export, evaluates a local SLO smoke check with dpone ops slo-evaluate, indexes the evidence, and uploads observability-maturity-report.

Published artifact:

  • observability-maturity-report

Files include:

  • run_report.json
  • prometheus_metrics.prom
  • opentelemetry_metrics.json
  • runtime_metrics.json
  • runtime_metrics.md
  • metrics_index.json
  • slo_report.json
  • artifact_index.json

Local reproduction:

uv run pytest tests/test_observability.py -q
uv run dpone observability metrics-export \
  --run-report test_artifacts/observability/maturity/run_report.json \
  --output-dir test_artifacts/observability/maturity/export \
  --label env=ci \
  --resource-attr deployment.environment=ci

Runbook: Observability maturity failures.

Full certification workflow

Workflow: .github/workflows/full-certification.yml.

This workflow is the recurring full source -> sink certification automation. It runs weekly on schedule and can also be started manually with a selected profile.

Profiles:

Profile Purpose Credentials
mock_contract Credential-free certification heartbeat for source -> sink contracts. no
mock_local Disposable local services when Docker/tooling is available. no external vendor credentials
vendor_live Managed/live vendor confidence when secrets are configured. yes

Automation flow:

flowchart TD
    Plan["certification-automation-plan"] --> Matrix["source/sink matrix"]
    Matrix --> MatrixReport["integration-matrix-report"]
    MatrixReport --> Benchmark["benchmark-baseline"]
    Benchmark --> Registry["run-registry"]
    Registry --> Lineage["lineage-export"]
    Lineage --> Evidence["evidence-bundle"]
    Evidence --> Strategy["strategy certification bundle"]
    Strategy --> Suite["certification-suite"]
    Suite --> Index["artifact-index"]
    Index --> Chain["evidence-chain"]
    Chain --> Verify["evidence-chain-verify"]
    Verify --> Artifact["full-certification-report"]

Published artifact:

  • full-certification-report

Local reproduction for the credential-free heartbeat:

DPONE_RUN_INTEGRATION=1 \
DPONE_RUN_INTEGRATION_MATRIX=1 \
DPONE_MATRIX_RUN_MODE=mock_contract \
DPONE_MATRIX_ROW_COUNT=10000 \
DPONE_MATRIX_ARTIFACT_DIR=test_artifacts/full_certification/matrix \
uv run pytest -m integration_matrix tests/integration/matrix -q

Runbook: Full certification automation failures.

Production maturity workflow

Workflow: .github/workflows/production-maturity.yml

Purpose: aggregate certification, CDC, performance, security, supply-chain, governance, and docs evidence into the final release-readiness artifact.

Primary command:

uv run dpone ops production-maturity \
  --release production-maturity-${GITHUB_RUN_ID} \
  --output-dir test_artifacts/production_maturity/report \
  --artifact certification=test_artifacts/production_maturity/input/certification.json \
  --artifact cdc=test_artifacts/production_maturity/input/cdc.json \
  --artifact performance=test_artifacts/production_maturity/input/performance.json \
  --artifact security=test_artifacts/production_maturity/input/security.json \
  --artifact supply_chain=test_artifacts/production_maturity/input/supply_chain.json \
  --artifact governance=test_artifacts/production_maturity/input/governance.json \
  --artifact docs=test_artifacts/production_maturity/input/docs.json

Artifacts:

Artifact Contents
production-maturity-report Input evidence, production_maturity.json, production_maturity.md, and artifact index.

Run this workflow weekly and before release promotion. For release candidates, replace the deterministic local evidence stubs with real artifacts from connector certification, CDC replay, benchmark, security, supply-chain, governance, and documentation workflows.

Industrial readiness workflow

Workflow: .github/workflows/industrial-readiness.yml

Purpose: aggregate local matrix, correctness, reliability, performance lab, UX, and governance evidence into the next industrial maturity gate.

Primary command:

uv run dpone ops industrial-readiness \
  --release industrial-readiness-${GITHUB_RUN_ID} \
  --output-dir test_artifacts/industrial_readiness/report \
  --artifact local_matrix=test_artifacts/industrial_readiness/input/local_matrix.json \
  --artifact correctness=test_artifacts/industrial_readiness/input/correctness.json \
  --artifact reliability=test_artifacts/industrial_readiness/input/reliability.json \
  --artifact performance_lab=test_artifacts/industrial_readiness/input/performance_lab.json \
  --artifact ux=test_artifacts/industrial_readiness/input/ux.json \
  --artifact governance=test_artifacts/industrial_readiness/input/governance.json

Artifacts:

Artifact Contents
industrial-readiness-report Input evidence, industrial_readiness.json, industrial_readiness.md, and artifact index.

Use this workflow after specialized matrix/correctness/reliability/performance/UX/governance evidence exists for a release candidate.