Data Cleaning Across Postgres, Duckdb, and PySpark

Background

If you work across Spark, DuckDB, and Postgres you've probably rewritten the same datetime or phone number cleaning logic three different ways. Most solutions either lock you into a package dependency or fall apart when you switch engines.

What it does

It's a copy-to-own framework for data cleaning (think shadcn but for data cleaning) that handles messy strings, datetimes, phone numbers. You pull the primitives into your own codebase instead of installing a package, so no dependency headaches. Under the hood it uses sqlframe to compile databricks-style syntax down to pyspark, duckdb, or postgres. Same cleaning logic, runs on all three.

Think of a multimodal pyjanitor that is significantly more flexible and powerful.

Target audience

Data engineers, analysts, and scientists who have to do data cleaning in Postgres or Spark or DuckDB. Been using it in production for a while, datetime stuff in particular has been solid.

How it differs from other tools

I know the obvious response is "just use claude code lol" and honestly fair, but I find AI-generated transformation code kind of hard to audit and debug when something goes wrong at scale. This is more for people who want something deterministic and reviewable that they actually own.

Try it

github: github.com/datacompose/datacompose | pip install datacompose | datacompose.io