Runtime Docker image¶

Use the runtime image when you want a repeatable container that can run dpone with the native tools required by high-throughput database paths.

The image is intentionally a CLI/runtime image, not a database server image. Use docker/docker-compose.integration.yml when you need local Postgres, MSSQL, ClickHouse, Kafka, Schema Registry and MinIO services for integration tests.

Included tools¶

Tool	Purpose
`dpone[full]`	All public Python connector extras.
Microsoft ODBC Driver 18	SQL Server connectivity through `pyodbc`.
`bcp` and `sqlcmd` from `mssql-tools18`	SQL Server bulk import/export and diagnostics.
`clickhouse-client`	Native ClickHouse TSV/client fast paths.
`unixODBC` headers/runtime	ODBC driver manager and build support.

The Dockerfile follows the official Microsoft Linux ODBC/tooling repository setup and the official ClickHouse Debian repository setup.

Build from PyPI¶

docker build \
  -f docker/runtime/Dockerfile \
  --build-arg DPONE_PACKAGE_SPEC='dpone[full]==X.Y.Z' \
  -t dpone:X.Y.Z .

Smoke it:

docker run --rm dpone:X.Y.Z --version
docker run --rm --entrypoint bcp dpone:X.Y.Z -v
docker run --rm --entrypoint clickhouse-client dpone:X.Y.Z --version

Pull from GHCR¶

Release tags publish the production runtime image to GitHub Container Registry:

docker pull ghcr.io/paulkov/dpone-runtime:X.Y.Z
docker run --rm ghcr.io/paulkov/dpone-runtime:X.Y.Z --version
docker run --rm --entrypoint bcp ghcr.io/paulkov/dpone-runtime:X.Y.Z -v
docker run --rm --entrypoint sqlcmd ghcr.io/paulkov/dpone-runtime:X.Y.Z -?
docker run --rm --entrypoint clickhouse-client ghcr.io/paulkov/dpone-runtime:X.Y.Z --version

Tag policy:

Tag	Meaning
`X.Y.Z`	Immutable runtime image for a dpone package release.
`sha-<commit>`	Traceability tag for the exact source commit.
`latest`	Latest pushed SemVer release tag. Do not use it for pinned production jobs.

Use the immutable version tag in production schedulers and CI. latest is useful for local exploration only.

Build from a direct wheel or private index¶

Use this mode when PyPI simple-index propagation is delayed but the wheel artifact is already available through a direct files.pythonhosted.org URL, a GitHub Release asset, or a private package index.

docker build \
  -f docker/runtime/Dockerfile \
  --build-arg DPONE_PACKAGE_SPEC='dpone[full] @ https://files.pythonhosted.org/.../dpone-X.Y.Z-py3-none-any.whl' \
  -t dpone:X.Y.Z-local .

Production images should normally pin immutable PyPI versions or a private package index artifact. Direct wheel URLs are best kept for release smoke tests and incident workarounds.

Run a manifest¶

docker run --rm \
  -v "$PWD:/workspace" \
  --env-file .env \
  ghcr.io/paulkov/dpone-runtime:X.Y.Z \
  run /workspace/examples/batch/landing_postgres_to_mssql.batch.yaml

For private registries, mirror the image into your internal registry and keep the same version tag. Do not rebuild a mutable image with the same tag and different package contents.

CI/CD workflow¶

The Runtime image GitHub Actions workflow builds the image for pull requests and manual runs. On release tags it waits for PyPI resolver visibility, builds the image from dpone[full]==X.Y.Z, smoke-tests dpone, bcp, sqlcmd and clickhouse-client, then pushes the version, sha-<commit> and latest tags to GHCR.

Manual dispatch supports:

version: package version to install;
package_spec: direct wheel/private index override;
push_image: push the image to GHCR for an explicit manual run.

Tuning notes¶

MSSQL fast paths use bcp; configure sink.options.bulk.bcp.* in manifests.
ClickHouse fast paths use HTTP/client TSV ingest; configure source.options.partitioning.* and the canonical native ingest settings shown by dpone plan.
Keep database credentials in environment variables, Vault, Airflow connections, or mounted secrets. Do not bake credentials into the image.
Use Linux runners for production images. macOS/Homebrew native tools are good for local testing, but Linux containers are easier to reproduce in CI.

Troubleshooting¶

Symptom	Check
`pyodbc` cannot find the driver	`docker run --rm --entrypoint odbcinst dpone:X.Y.Z -q -d`
`bcp: command not found`	Confirm `/opt/mssql-tools18/bin` is in `PATH`.
ClickHouse client missing	Rebuild and check the ClickHouse repository step.
`pip install dpone==X.Y.Z` fails during image build	Run `python tools/pypi_release_smoke.py --package dpone --version X.Y.Z --install-smoke` and wait for PyPI simple-index visibility, or build from a direct wheel/private index.