Skip to content

Testing documentation

This folder is the single home for dpone testing documentation. Start here when you need to choose a test layer, run a gate, debug a red workflow, or understand the source -> sink certification matrix.

Documentation map

Need Doc
Local/default regression gate, package smoke, generated docs checks Local/default testing
Service markers, live/vendor boundaries, Docker services, artifact policy Integration tests
Replay recovery/control-plane gate for resync/resume adapters Replay integration
Manual source -> sink matrix category and GitHub Actions workflow Manual integration matrix
Manual local-live/vendor-live connector certification and benchmark/SLO evidence Live certification
Detailed 200-case source -> sink strategy behavior model, volumes, artifacts, failure recovery Source/sink matrix runbook
SQL Server local wide-table mock matrix and MSSQL-specific troubleshooting Local MSSQL mock matrix
Local native transfer benchmarks for Postgres -> MSSQL and MSSQL -> ClickHouse Native transfer benchmark artifact: 2026-06-09
Connector capability evidence Connector certification
CI/CD workflow map and red-build runbooks CI/CD

Quick decision guide

flowchart TD
    A["What changed?"] --> B{"Runtime code or public contract?"}
    B -- yes --> C["Run default regression gate"]
    B -- docs only --> D["Run docs contract tests and mkdocs build"]
    C --> E{"Connector, strategy, or source/sink behavior?"}
    E -- yes --> F["Run mock_contract matrix"]
    E -- local DB/Kafka path --> G["Run mock_local or service marker"]
    E -- vendor/API path --> H["Run vendor_live certification"]
    D --> I["Update docs links/nav/index if needed"]

Test layers

Layer Marker / workflow External credentials Purpose
Unit/contract default pytest no Fast local confidence for models, planners, codecs, SQL builders, docs contracts, CLI output.
Default non-live gate pytest -m "not integration_live" no Release-quality local gate for ordinary changes.
mock_contract matrix integration_matrix no Validates all 25 source -> sink pairs and 200 strategy contracts, including snapshot_diff, DB-target partition_replace/scd2/backfill, Postgres xmin, and Postgres/MSSQL cdc, without starting services.
mock_local matrix integration_matrix_mock no Starts disposable local services where possible and validates local/mock pipelines.
Service-specific integration integration_postgres, integration_mssql, integration_clickhouse, integration_kafka no for local Docker, yes for external endpoints Exercises native clients, staging, bulk paths, state, CDC, and connector behavior.
local_live certification live-certification.yml no external credentials Starts local Postgres/MSSQL/ClickHouse/Kafka/MinIO, runs service markers, matrix, benchmark-slo-gate, certification pack, and evidence chain.
real_local certification live-certification.yml no external credentials Runs the local service stack with DPONE_MATRIX_RUN_MODE=real_local and produces performance, state/reconciliation, release evidence pack, and evidence chain artifacts for minor/major releases.
Replay integration integration_replay no by default Validates replay adapter ordering, injected live-backend contracts, reconciliation, and state commit behavior for dpone resync and dpone resume.
vendor_live integration_live and provider markers yes Exercises real managed/vendor systems such as BigQuery or external APIs.

Credentials are only required for vendor_live. The mock_contract, mock_local, and real_local layers use deterministic local test passwords, local containers, or in-process mock servers.

Where tests live

Area Path
Source -> sink strategy matrix tests/integration/matrix/
Matrix registry contracts tests/test_integration_matrix_contracts.py
CI/CD documentation contracts tests/test_cicd_docs_contracts.py
Docs and language contracts tests/test_docs_*
PostgreSQL live/local tests tests/integration/postgres/
MSSQL live/local tests tests/integration/mssql/
ClickHouse live/local tests tests/integration/clickhouse/
Kafka live/local tests tests/integration/kafka/
Replay recovery/control-plane tests tests/integration/replay/
Provider/API mock and live tests tests/integration/<provider>/

Common commands

Default release-quality gate:

uv sync --all-extras
uv run ruff check .
uv run ruff format --check .
uv run mypy --config-file mypy.ini
uv run pytest -m "not integration_live" --cov=src/dpone --cov-report=xml
uv build

Docs-only gate:

uv run pytest tests/test_cicd_docs_contracts.py tests/test_docs_mermaid_contracts.py -q
python -m pip install -r docs/requirements.txt
mkdocs build --strict

Credential-free source/sink matrix:

DPONE_RUN_INTEGRATION=1 \
DPONE_RUN_INTEGRATION_MATRIX=1 \
DPONE_MATRIX_RUN_MODE=mock_contract \
DPONE_MATRIX_ARTIFACT_DIR=test_artifacts/integration_matrix/mock_contract_latest \
uv run pytest -m integration_matrix tests/integration/matrix -q

Local/mock source/sink matrix:

DPONE_RUN_INTEGRATION=1 \
DPONE_RUN_INTEGRATION_MATRIX=1 \
DPONE_MATRIX_RUN_MODE=mock_local \
DPONE_MATRIX_ARTIFACT_DIR=test_artifacts/integration_matrix/mock_local_latest \
uv run pytest -m integration_matrix_mock tests/integration/matrix -q

Replay recovery/control-plane gate:

DPONE_RUN_INTEGRATION_REPLAY=1 uv run pytest -m integration_replay tests/integration/replay -q

Matrix behavior artifacts

When DPONE_MATRIX_ARTIFACT_DIR is set, the matrix writes one metadata artifact and one strategy behavior artifact per selected case. Each behavior artifact uses a configurable deterministic volume profile: 10,000 base source rows by default, 20% changed rows, 5% physical deletes, and 120 wide_* source columns with mixed null sparsity.

Use the __behavior.json artifact to compare full-volume counts/checksums first, then inspect target_before, source_rows, expected_rows, and actual_rows samples when a strategy case turns red.

Benchmark artifacts

Use benchmark artifacts when a change touches native bulk transfer, partitioning, or target ingest performance:

Runbook: choose the right layer

  1. Use default pytest for every PR.
  2. Use mock_contract when docs, manifests, strategy support, or source -> sink coverage changes.
  3. Use mock_local when connector runtime, staging, type mapping, or load strategy behavior changes.
  4. Use service-specific markers when native clients or local DB/Kafka behavior changed.
  5. Use vendor_live before release or after changes that affect managed external systems.

Runbook: common failures

Failure: mock_contract fails.

Failure: mock_local service does not start.

  • Re-run docker compose -f docker/docker-compose.integration.yml ps.
  • Inspect logs for the failing service with docker logs <container>.
  • For MSSQL, verify the password satisfies SQL Server complexity rules and that the runner has enough memory.
  • For Kafka, verify kafka becomes healthy before schema-registry starts.

Failure: vendor_live fails.

  • Verify credentials are present in GitHub Actions secrets or local environment.
  • Re-run a focused case with DPONE_MATRIX_CASE_ID=<case> when the matrix is involved.
  • Never copy credentials into artifacts or logs.

Failure: docs links break after moving testing pages.

  • Keep testing pages under docs/testing/.
  • Update mkdocs.yml, Documentation index, and README links together.
  • Run mkdocs build --strict before pushing.

Nested normalization evidence