Skip to content

Connector certification

Connector certification defines whether a source or sink is experimental, usable in production with constraints, or production-certified.

Certification levels

Experimental

Use for connectors that are useful for development but not guaranteed for production.

Requirements:

  • Importable without optional dependency leakage.
  • Public docs mention required extras.
  • Basic configuration examples exist.
  • Offline construction tests exist for requests, SQL, or client options.

Beta

Use for connectors that are suitable for controlled workloads.

Requirements:

  • All experimental requirements.
  • Credential loading tests with fake providers.
  • Retry/rate-limit behavior documented.
  • Data type mapping tests.
  • Incremental or pagination tests where applicable.
  • Local integration test when a disposable service is practical.

Production-certified

Use for connectors approved for unattended production pipelines.

Requirements:

  • All beta requirements.
  • Live integration test profile behind integration_live.
  • Idempotent rerun behavior documented and tested.
  • Checkpoint recovery tested.
  • Schema drift behavior documented.
  • Failure taxonomy documented.
  • Observability fields verified.
  • Backward compatibility policy documented.

Connector matrix

Connector Type Current level Next step
PostgreSQL source/sink/state/CDC Beta Promote after recurring CDC/XMin checkpoint recovery evidence is green for several scheduled runs.
MSSQL source/sink/state/CDC Beta Promote after recurring Docker or managed SQL Server bcp/CDC certification runs are green.
ClickHouse source/sink Beta Promote after recurring staging-first sink and direct TSV/HTTP certification runs are green.
Kafka bounded batch source/sink/state offsets Beta Promote after recurring Kafka + Schema Registry format matrix runs are green.
REST API generic source Beta Promote after pagination/auth/incremental contract matrix is expanded with additional provider fixtures.
BigQuery sink/state Beta Add live BigQuery integration profile and table lifecycle tests.
Google Ads source Experimental Add fake API client tests and live opt-in profile.
Google Sheets source Experimental Add fake API client tests and live opt-in profile.
AppsFlyer source Experimental Add fake HTTP pagination/rate-limit tests.
CBR source Experimental Add fake HTTP contract tests.
FastTrack source Experimental Add fake HTTP pagination tests.
Mindbox source Experimental Add fake HTTP retry/rate-limit tests.
Omnidesk source Experimental Add fake HTTP pagination and incremental tests.
OpenExchangeRates source Experimental Add fake HTTP response contract tests.
Similarweb source Experimental Add fake HTTP contract tests.
Yandex Webmaster source Beta Add live opt-in profile and rate-limit tests.

Required test markers

  • unit: default tests that require no external services.
  • integration: local disposable infrastructure.
  • integration_live: real vendor services and real credentials.
  • nightly: scheduled broad validation.

GitHub Actions gates

The public OSS repository uses two certification layers:

  • .github/workflows/ci.yml is the required pull request quality gate for non-live checks.
  • .github/workflows/connector-certification.yml is the recurring certification gate.

The recurring gate runs on a daily schedule and can also be started manually from GitHub Actions.

It produces:

  • connector-certification-offline: the dpone certify matrix in JSON and Markdown.
  • local live connector results for disposable or preconfigured Postgres, MSSQL, ClickHouse, Kafka and Postgres CDC markers.
  • vendor live results when the corresponding GitHub secrets are configured.

Live certification is intentionally opt-in at the test level: if an external system or credential is not configured, that connector's integration test reports a skip rather than silently using fake data.

Certification checklist

For each connector, add or update:

  • Documentation page.
  • Example manifest.
  • Credential model.
  • Offline unit tests.
  • Optional dependency smoke test.
  • Integration test profile.
  • Live test instructions.
  • Known limitations.
  • Changelog entry when level changes.