Skip to content

Source -> Sink Matrix

This matrix is the public entrypoint for supported dpone source -> sink documentation. Each row links to a dedicated guide with install extras, manifest shape, supported strategies, staging policy, schema evolution behavior, delete/reconciliation notes, and type mapping policy.

Start from your route (source → sink): after you pick a matrix row for Airflow self-service, scaffold it with a built-in recipe on First Airflow DAG.

Use Route certification pack and Route readiness when a supported row needs a machine-readable go/no-go report for a specific strategy and evidence bundle. Use Route certification matrix when support must be stated for the exact transport, schema-evolution mode, and Airflow/runtime mode. A source/sink row alone is not a production claim. Use Route execution ledger when the same route also needs idempotent retry, lease fencing, resume, repair, resync, or state-commit evidence. Use Route state promotion when a route must advance source state only after a matching sink commit receipt and route execution ledger. Use Route live certification when Docker-live or vendor-live artifacts must be normalized into route_live_evidence_bundle before release review. Use Route certify when refresh execute, snapshot capture, exact verification, readiness, and release evidence must become one route_certification_bundle. Use Route certify release when all first-class route bundles must be aggregated into a release-level go/no-go report with route_certification_release.json. Use Route release finalize when release managers need bundle discovery, freshness, provenance, regression, and route history checks before tagging. Use Route release candidate orchestrator when the whole route release train must be captured in one route_rc_orchestration receipt. Use Route release candidate executor when that receipt must be dry-run checked or explicitly executed into a route_rc_execution receipt. Use Route release gate when those artifacts must be combined into one release-candidate go/no-go receipt. Use Route Conformance Lab when postgres -> mssql or mssql -> clickhouse needs 10,000-row, 200-column --adapter vendor_live or --adapter docker evidence before release.

Visual overview

flowchart LR
    subgraph Sources["Batch sources"]
        PG["PostgreSQL"]
        MS["MSSQL / SQL Server"]
        MY["MySQL"]
        CH["ClickHouse"]
        API["Generic REST API"]
        KIN["Kafka bounded topic"]
    end

    subgraph Runtime["dpone runtime contracts"]
        Extract["ExtractResult + schema"]
        Plan["Schema evolution + plan"]
        Stage["Staging or shadow table"]
        Commit["Set-based commit + state"]
    end

    subgraph Sinks["Targets"]
        MSS["MSSQL / SQL Server"]
        PGS["PostgreSQL"]
        CHS["ClickHouse"]
        BQ["BigQuery"]
        KOUT["Kafka topic"]
    end

    PG --> Extract
    MS --> Extract
    MY --> Extract
    CH --> Extract
    API --> Extract
    KIN --> Extract
    Extract --> Plan --> Stage --> Commit
    Commit --> MSS
    Commit --> PGS
    Commit --> CHS
    Commit --> BQ
    Commit --> KOUT

Supported source and sink families

Sources:

  • PostgreSQL
  • MSSQL / SQL Server
  • MySQL
  • ClickHouse
  • Generic REST API
  • Kafka bounded batch topic

Sinks:

  • MSSQL / SQL Server
  • PostgreSQL
  • ClickHouse
  • BigQuery
  • Kafka topic

Matrix

Source Sink Guide Status Install
PostgreSQL MSSQL / SQL Server postgres -> mssql Batch ETL supported (wide vendor-live for FR/append/merge/replace/partition_replace/snapshot_diff/scd2/backfill; bytea via hex character BCP) pip install "dpone[mssql,postgres]"
MySQL MSSQL / SQL Server mysql -> mssql Batch ETL supported (wide vendor-live; BLOB via hex character BCP) pip install "dpone[mysql,mssql]"
MySQL PostgreSQL mysql -> postgres Batch ETL supported pip install "dpone[mysql,postgres]"
MySQL ClickHouse mysql -> clickhouse Batch ETL supported (wide vendor-live for FR/append/merge/replace/partition_replace/snapshot_diff/scd2/backfill) pip install "dpone[mysql,clickhouse]"
MySQL BigQuery mysql -> bigquery Batch ETL supported pip install "dpone[mysql,gcp]"
MySQL Kafka topic mysql -> kafka Batch/event-log supported pip install "dpone[mysql,kafka]"
PostgreSQL PostgreSQL postgres -> postgres Batch ETL supported (wide vendor-live for FR/append/merge/replace/partition_replace/snapshot_diff/scd2/backfill) pip install "dpone[postgres]"
PostgreSQL ClickHouse postgres -> clickhouse Batch ETL supported (wide vendor-live for FR/append/merge/replace/partition_replace/snapshot_diff/scd2/backfill) pip install "dpone[clickhouse,postgres]"
PostgreSQL BigQuery postgres -> bigquery Batch ETL supported (wide vendor-live for FR/append/merge/replace/partition_replace/snapshot_diff/scd2/backfill) pip install "dpone[gcp,postgres]"
PostgreSQL Kafka topic postgres -> kafka Batch/event-log supported (wide vendor-live for FR/append/merge/replace/snapshot_diff/backfill-merge; partition_replace/scd2 N/A) pip install "dpone[kafka,postgres]"
MSSQL / SQL Server MSSQL / SQL Server mssql -> mssql Batch ETL supported (wide vendor-live for FR/append/merge/replace/partition_replace/snapshot_diff/scd2/backfill; varbinary via hex character BCP) pip install "dpone[mssql]"
MSSQL / SQL Server PostgreSQL mssql -> postgres Batch ETL supported (wide vendor-live for FR/append/merge/replace/partition_replace/snapshot_diff/scd2/backfill; CSV COPY via mssql_to_postgres_native_v1) pip install "dpone[mssql,postgres]"
MSSQL / SQL Server ClickHouse mssql -> clickhouse Batch ETL supported (wide vendor-live for FR/append/merge/replace/partition_replace/snapshot_diff/scd2/backfill; varbinary via hex character BCP); live CDC runtime adapters, CDC poison quarantine, replay dedupe, CDC compare and repair, CDC retention gap auto-resync, CDC serving materialization, typed schema drift checks, and parse quarantine profiled pip install "dpone[clickhouse,mssql]"
MSSQL / SQL Server BigQuery mssql -> bigquery Batch ETL supported (wide vendor-live for FR/append/merge/replace/partition_replace/snapshot_diff/scd2/backfill; mssql_to_bigquery_analytics_v1) pip install "dpone[gcp,mssql]"
MSSQL / SQL Server Kafka topic mssql -> kafka Batch/event-log supported (wide vendor-live for FR/append/merge/replace/snapshot_diff/backfill-merge; partition_replace/scd2 N/A) pip install "dpone[kafka,mssql]"
ClickHouse MSSQL / SQL Server clickhouse -> mssql Governed batch ETL supported; certified type profile, MSSQL bcp staging, event-boundary advisor, route readiness evidence, and source/target acceptance required for production scheduling pip install "dpone[clickhouse,mssql]"
ClickHouse PostgreSQL clickhouse -> postgres Batch ETL supported pip install "dpone[clickhouse,postgres]"
ClickHouse ClickHouse clickhouse -> clickhouse Batch ETL supported pip install "dpone[clickhouse]"
ClickHouse BigQuery clickhouse -> bigquery Batch ETL supported pip install "dpone[clickhouse,gcp]"
ClickHouse Kafka topic clickhouse -> kafka Batch/event-log supported pip install "dpone[clickhouse,kafka]"
Generic REST API MSSQL / SQL Server api -> mssql Batch ETL supported pip install "dpone[mssql]"
Generic REST API PostgreSQL api -> postgres Batch ETL supported pip install "dpone[postgres]"
Generic REST API ClickHouse api -> clickhouse Batch ETL supported pip install "dpone[clickhouse]"
Generic REST API BigQuery api -> bigquery Batch ETL supported pip install "dpone[gcp]"
Generic REST API Kafka topic api -> kafka Batch/event-log supported pip install "dpone[kafka]"
Kafka batch topic MSSQL / SQL Server kafka -> mssql Batch/event-log supported pip install "dpone[kafka,mssql]"
Kafka batch topic PostgreSQL kafka -> postgres Batch/event-log supported pip install "dpone[kafka,postgres]"
Kafka batch topic ClickHouse kafka -> clickhouse Batch/event-log supported pip install "dpone[clickhouse,kafka]"
Kafka batch topic BigQuery kafka -> bigquery Batch/event-log supported pip install "dpone[gcp,kafka]"
Kafka batch topic Kafka topic kafka -> kafka Batch/event-log supported pip install "dpone[kafka]"

Production rules that apply to every guide

  • Loads are staging-first. Heavy operations must happen in staging/shadow tables, not directly in final target tables.
  • Automatic schema evolution is enabled by default and fail-closed for breaking changes.
  • Type conversion must be explicit when vendor-specific types are involved.
  • State is advanced only after sink commit succeeds.
  • Physical deletes require CDC/tombstones or snapshot reconciliation; they are never inferred silently from incremental cursors.
  • Kafka is batch ETL/event-log integration in this release, not an infinite streaming runtime.

Type mapping

Use Type mapping matrix for cross-system type conversion policy and per-family caveats.