Skip to content

Testing

dpone separates fast regression tests from local service integration tests and vendor live certification.

Default regression gate

Run this gate before publishing or merging runtime changes:

uv sync --all-extras
uv run ruff check .
uv run ruff format --check .
uv run mypy --config-file mypy.ini
uv run pytest -m "not integration_live"
uv build

Targeted unit and contract tests

Use targeted tests while developing a specific subsystem:

uv run pytest tests/test_runtime_schema_evolution_contracts.py -q
uv run pytest tests/test_runtime_kafka_contracts.py -q
uv run pytest tests/test_docs_language_contracts.py -q

Manual integration matrix

The matrix has two credential-free layers:

  • mock_contract validates 200 source -> sink -> strategy contracts: 100 common base cases plus 25 snapshot_diff cases plus 60 DB-target partition_replace/scd2/backfill cases plus Postgres xmin and Postgres/MSSQL cdc source-specific cases. It writes deterministic metadata/behavior artifacts using 10,000 base rows by default, 20% changed rows, 5% physical deletes, and 120 sparse wide source columns per sampled row.
  • mock_local runs local/mock-capable cases and skips documented-only BigQuery targets until a managed vendor_live run is used.

The full source/sink matrix is intentionally manual because it can start local databases, Kafka, Schema Registry, and wide-table fixtures.

uv run pytest tests/integration/matrix -m integration_matrix_mock

See Manual integration matrix and Testing runbooks.

Live vendor tests

Vendor tests are opt-in and should not block normal pull requests unless the connector is being certified.

DPONE_RUN_INTEGRATION=1 \
DPONE_RUN_INTEGRATION_LIVE=1 \
uv run pytest tests/integration -m integration_live

Use live tests for connector certification, release candidates, and scheduled confidence checks.

Generated documentation checks

Compatibility and metrics documentation may contain generated sections. Regenerate them with the dedicated tool before committing changes that alter supported APIs or compatibility contracts.

uv run dpone docs check-compatibility
uv run dpone docs update-dev-metrics --check

Snapshot artifacts

Long-running benchmarks and certification runs should write artifacts under test_artifacts/ with a unique date, runner, environment, command, and result summary.

For normal local runs, runtime reports are written under .dpone/runs/ unless the manifest overrides the artifact path.