Source -> Sink Matrix¶

This matrix is the public entrypoint for supported dpone source -> sink documentation. Each row links to a dedicated guide with install extras, manifest shape, supported strategies, staging policy, schema evolution behavior, delete/reconciliation notes, and type mapping policy.

Visual overview¶

flowchart LR
    subgraph Sources["Batch sources"]
        PG["PostgreSQL"]
        MS["MSSQL / SQL Server"]
        CH["ClickHouse"]
        API["Generic REST API"]
        KIN["Kafka bounded topic"]
    end

    subgraph Runtime["dpone runtime contracts"]
        Extract["ExtractResult + schema"]
        Plan["Schema evolution + plan"]
        Stage["Staging or shadow table"]
        Commit["Set-based commit + state"]
    end

    subgraph Sinks["Targets"]
        MSS["MSSQL / SQL Server"]
        PGS["PostgreSQL"]
        CHS["ClickHouse"]
        BQ["BigQuery"]
        KOUT["Kafka topic"]
    end

    PG --> Extract
    MS --> Extract
    CH --> Extract
    API --> Extract
    KIN --> Extract
    Extract --> Plan --> Stage --> Commit
    Commit --> MSS
    Commit --> PGS
    Commit --> CHS
    Commit --> BQ
    Commit --> KOUT

Supported source and sink families¶

Sources:

PostgreSQL
MSSQL / SQL Server
ClickHouse
Generic REST API
Kafka bounded batch topic

Sinks:

MSSQL / SQL Server
PostgreSQL
ClickHouse
BigQuery
Kafka topic

Matrix¶

Source	Sink	Guide	Status	Install
PostgreSQL	MSSQL / SQL Server	postgres -> mssql	Batch ETL supported	pip install "dpone[mssql,postgres]"
PostgreSQL	PostgreSQL	postgres -> postgres	Batch ETL supported	pip install "dpone[postgres]"
PostgreSQL	ClickHouse	postgres -> clickhouse	Batch ETL supported	pip install "dpone[clickhouse,postgres]"
PostgreSQL	BigQuery	postgres -> bigquery	Batch ETL supported	pip install "dpone[gcp,postgres]"
PostgreSQL	Kafka topic	postgres -> kafka	Batch/event-log supported	pip install "dpone[kafka,postgres]"
MSSQL / SQL Server	MSSQL / SQL Server	mssql -> mssql	Batch ETL supported	pip install "dpone[mssql]"
MSSQL / SQL Server	PostgreSQL	mssql -> postgres	Batch ETL supported	pip install "dpone[mssql,postgres]"
MSSQL / SQL Server	ClickHouse	mssql -> clickhouse	Batch ETL supported	pip install "dpone[clickhouse,mssql]"
MSSQL / SQL Server	BigQuery	mssql -> bigquery	Batch ETL supported	pip install "dpone[gcp,mssql]"
MSSQL / SQL Server	Kafka topic	mssql -> kafka	Batch/event-log supported	pip install "dpone[kafka,mssql]"
ClickHouse	MSSQL / SQL Server	clickhouse -> mssql	Batch ETL supported	pip install "dpone[clickhouse,mssql]"
ClickHouse	PostgreSQL	clickhouse -> postgres	Batch ETL supported	pip install "dpone[clickhouse,postgres]"
ClickHouse	ClickHouse	clickhouse -> clickhouse	Batch ETL supported	pip install "dpone[clickhouse]"
ClickHouse	BigQuery	clickhouse -> bigquery	Batch ETL supported	pip install "dpone[clickhouse,gcp]"
ClickHouse	Kafka topic	clickhouse -> kafka	Batch/event-log supported	pip install "dpone[clickhouse,kafka]"
Generic REST API	MSSQL / SQL Server	api -> mssql	Batch ETL supported	pip install "dpone[mssql]"
Generic REST API	PostgreSQL	api -> postgres	Batch ETL supported	pip install "dpone[postgres]"
Generic REST API	ClickHouse	api -> clickhouse	Batch ETL supported	pip install "dpone[clickhouse]"
Generic REST API	BigQuery	api -> bigquery	Batch ETL supported	pip install "dpone[gcp]"
Generic REST API	Kafka topic	api -> kafka	Batch/event-log supported	pip install "dpone[kafka]"
Kafka batch topic	MSSQL / SQL Server	kafka -> mssql	Batch/event-log supported	pip install "dpone[kafka,mssql]"
Kafka batch topic	PostgreSQL	kafka -> postgres	Batch/event-log supported	pip install "dpone[kafka,postgres]"
Kafka batch topic	ClickHouse	kafka -> clickhouse	Batch/event-log supported	pip install "dpone[clickhouse,kafka]"
Kafka batch topic	BigQuery	kafka -> bigquery	Batch/event-log supported	pip install "dpone[gcp,kafka]"
Kafka batch topic	Kafka topic	kafka -> kafka	Batch/event-log supported	pip install "dpone[kafka]"

Production rules that apply to every guide¶

Loads are staging-first. Heavy operations must happen in staging/shadow tables, not directly in final target tables.
Automatic schema evolution is enabled by default and fail-closed for breaking changes.
Type conversion must be explicit when vendor-specific types are involved.
State is advanced only after sink commit succeeds.
Physical deletes require CDC/tombstones or snapshot reconciliation; they are never inferred silently from incremental cursors.
Kafka is batch ETL/event-log integration in this release, not an infinite streaming runtime.

Type mapping¶

Use Type mapping matrix for cross-system type conversion policy and per-family caveats.

Schema evolution explains automatic target DDL and __dpone__nc__* generated columns.
Postgres XMin incremental strategy is the detailed runbook for Postgres transaction-ID based incremental extraction.
Type mapping matrix documents source -> sink type conversion policy.
Load strategies is the canonical guide for full_refresh, incremental_append, incremental_merge/upsert, replace, partition_replace, snapshot_diff, scd2, cdc_apply, and backfill semantics.