DataComPy is a package to compare two DataFrames (or tables) such as Pandas, Spark, Polars, and even Snowflake. Originally it was created to be something of a replacement for SAS's PROC COMPARE for Pandas DataFrames with some more functionality than just Pandas.DataFrame.equals(Pandas.DataFrame) (in that it prints out some stats, and lets you tweak how accurate matches have to be). Supported types include:

• Pandas
• Polars
• Spark
• Snowflake (via snowpark)
• Dask (via Fugue)
• DuckDB (via Fugue)

pip install datacompy

or

conda install datacompy

If you would like to use Spark or any other backends please make sure you install via extras:

pip install datacompy[spark]
pip install datacompy[fugue]
pip install datacompy[snowflake]

Starting with v0.17.0, both LegacySparkCompare and SparkPandasCompare have been removed.

Different versions of Spark, Pandas, and Python interact differently. Below is a matrix of what we test with. With the move to Pandas on Spark API and compatability issues with Pandas 2+ we will for the mean time note support Pandas 2 with the Pandas on Spark implementation. Spark plans to support Pandas 2 in Spark 4

• Pandas: (See documentation)
• Spark: (See documentation)
• Polars: (See documentation)
• Snowflake/Snowpark: (See documentation)
• Fugue is a Python library that provides a unified interface for data processing on Pandas, DuckDB, Polars, Arrow, Spark, Dask, Ray, and many other backends. DataComPy integrates with Fugue to provide a simple way to compare data across these backends. Please note that Fugue will use the Pandas (Native) logic at its lowest level (See documentation)

We welcome and appreciate your contributions! Before we can accept any contributions, we ask that you please be sure to sign the Contributor License Agreement (CLA).

This project adheres to the Open Source Code of Conduct. By participating, you are expected to honor this code.

Roadmap details can be found here