Overview

• fix publishing issues
• fix docs publishing issues

What's Changed

• fix: publish connect sources and javadocs by @SemyonSinchenko in #654
• feat: fix docs path by @SemyonSinchenko in #656
• feat: 0.9.2 release by @SemyonSinchenko in #658

Full Changelog: v0.9.1...v0.9.2

Overview

Mostly this release is about publishing fixes (version string, README, PySpark Classic dependencies, etc.). There are no changes in core or API.

What's Changed

• Fix versioning by @Kimahriman in #641
• bug: Let classic python version work without grpc deps by @Kimahriman in #647
• fix: resolve gf-connect is not published by @SemyonSinchenko in #645
• feat: prepare the 0.9.1 release by @SemyonSinchenko in #650

Full Changelog: v0.9.0...v0.9.1

Overview

• Spark Connect support
• Spark 4.x support
• Performance improvements in CC
• Updated API for Pregel
• GraphX-free impl of CDLP and SP
• New groupId io.graphframes
• New PyPi ID: graphframes-py
• A lot of additional changes and fixes
• Updated documentation

What's Changed

• Add JVM args to fix sbt.ForkMain error by @EnricoMi in #438
• Add Spark 3.5.0 support by @EnricoMi in #436
• Add log4j2.properties to test resources by @EnricoMi in #437
• Fix connected component by @WeichenXu123 in #454
• Update doc and releasing guide by @WeichenXu123 in #455
• Update quick-start.md to use latest build of GraphFrames jar by @rjurney in #461
• Java opt MaxPermSize --> MaxMetaspaceSize, includes for Java 17+ sun.nio.ch, java.lang ALL-UNNAMED by @rjurney in #464
• Upgrade github actions by @bjornjorgensen in #471
• Update to spark-version: 3.5.4 and fix Connected component suite by @bjornjorgensen in #478
• Add Type hints for Python code by @bjornjorgensen in #480
• Fix the bug I introduced previously when adapting build.sbt for JDK 11 by @rjurney in #476
• Update scala code by @bjornjorgensen in #479
• Use vertices SparkSession by @Nassizouz in #474
• Update .mailmap to include me by @rjurney in #482
• Update .mailmap to include rjurney by @rjurney in #483
• [DOCS] Adding new super cool logo to the docs by @SauronShepherd in #495
• adding favicon by @SauronShepherd in #497
• chore: drop sbt-spark-packages && update plugins by @SemyonSinchenko in #487
• Add Pull request template by @bjornjorgensen in #472
• doc: update badges in README by @SemyonSinchenko in #509
• chore: add scalafmt and re-format all the code by @SemyonSinchenko in #507
• 1 of 3: Build a graphframes Python package during the build process by @rjurney in #512
• Rjurney/motif tutorial code min by @rjurney in #520
• 3 of 3: Documentation cleanup and update. Added a motif finding tutorial. by @rjurney in #511
• chore: replace setup-scala action by @SemyonSinchenko in #522
• chore: suppress warnings about incomplete patterns by @SemyonSinchenko in #526
• chore: testing against both java-11 and java-17 by @SemyonSinchenko in #525
• chore: force python code style check in CI by @SemyonSinchenko in #527
• chore: pre-commit hook by @SemyonSinchenko in #534
• Convert nose / unittest tests to pytest by @rjurney in #524
• fixed a minor nit usage issue by @dmatrix in #535
• feat: power iteration clustering by @SemyonSinchenko in #529
• Add a missing image in preparation for docs release by @rjurney in #537
• feat: SparkConnect support by @SemyonSinchenko in #506
• [ConnectedComponents] Memory leak with unpersisted DataFrames in the last round by @SauronShepherd in #552
• feat: add early stopping to Pregel by @SemyonSinchenko in #550
• Changes the output label of CC to min of original ID by @architch in #320
• Quote column names with backticks by @EnricoMi in #370
• feat: add an experimental implementation of SP in GF by @SemyonSinchenko in #558
• feat: add LDBC tests by @SemyonSinchenko in #570
• chore: add "-Xlint" and scalafix checks && fix all errors && update pre-commit by @SemyonSinchenko in #575
• chore: automate docs publishing by @SemyonSinchenko in #566
• feat: add vertices "voting" ability and optional flags to Pregel by @SemyonSinchenko in #581
• feat: python publish by @SemyonSinchenko in #577
• feat: add a new mixin WithMaxIter and propogate it to GraphX CC by @SemyonSinchenko in #590
• Allow setting default ConnectedComponents configurations from spark config by @james-willis in #563
• Bump setuptools from 78.1.0 to 78.1.1 in /python by @dependabot[bot] in #596
• Removing unused file as per #585 by @Conor0Callaghan in #592
• feat: LabelPropagation without GraphX by @SemyonSinchenko in #587
• feat: sbt-ci-release by @SemyonSinchenko in #598
• Bump requests from 2.32.3 to 2.32.4 in /python by @dependabot[bot] in #601
• Bump protobuf from 5.29.4 to 5.29.5 in /python by @dependabot[bot] in #606
• Bump urllib3 from 2.4.0 to 2.5.0 in /python by @dependabot[bot] in #607
• feat: Scala publish CI GitHub Action by @SemyonSinchenko in #600
• feat: Spark 4.0 support by @Kimahriman in #608
• chore: Add issue templates by @SemyonSinchenko in #605
• chore: Cleanup assembly and shading by @Kimahriman in #617
• chore: fix shading problems and slightly update CI by @SemyonSinchenko in #615
• fix: try to fix publishing by @SemyonSinchenko in #619
• chore: add useLabelsAsComponents configuration for connected components by @SemyonSinchenko in #623
• doc: update README with details on nightly builds by @SemyonSinchenko in #622
• README and .gitignore update to add GraphFrames classes by @rjurney in #629
• allow useLabelsAsComponents to be set locally on the ConnectedComponents instance by @james-willis in #632
• feat: set plugin and update pyproject by @SemyonSinchenko in #634
• feat: planning release 0.9.0 by @SemyonSinchenko in #626
• fix: remove version.sbt by @SemyonSinchenko in #637

New Contributors

• @bjornjorgensen made their first contribution in #471
• @Nassizouz made their first contribution in #474
• @SauronShepherd made their first contribution in #495
• @SemyonSinchenko made their first contribution in #487
• @dmatrix made their first contribution in #535
• @architch made their first contribution in #320
• @james-willis made their first contribution in #563
• @dependabot[bot] made their first contribution in #596
• @Conor0Callaghan made their first contribution in #592
• @Kimahriman made their first contribution in #608

Full Changelog: v0.8.3...v0.9.0

Major Changes

• Support Spark 3.3 / Scala 2.12 , Spark 3.4 / Scala 2.12 and Scala 2.13, Spark 3.5 / Scala 2.12 and Scala 2.13
• Remove deprecated SQLContext from code

What's Changed

• Use official Spark 3.2 PyPI release in CI
• Upgrade to latest spark versions 3.3.0, 3.2.2 and 3.1.3
• Python: Replace SQLContext with SparkSession
• Update CI actions and dependencies
• Add support for scala 2.13 (
• Remove deprecated sqlContext in graphframe scala code
• Spark 3.4.1 and drop EOL Spark versions 3.0 and 3.1

New Contributors

• eejbyfeldt
• ttc0419

Full Changelog: v0.8.2...v0.8.3

Major Changes

• Support Spark 3.1 / Scala 2.12 and Spark 3.2 / Scala 2.12.

What's Changed

• [ML-12428] Upgrade CI spark version to 3.0.1 by @WeichenXu123 in #372
• Extend to Spark 3.1 by @EnricoMi in #381
• Correct the order of the analogy by @johnnyleitrim in #377
• Replace bintray by @menglewis in #385
• Switch from Travis CI to GitHub Actions by @mengxr in #402
• Build against Spark 3.2 by @mengxr in #400

New Contributors

• @EnricoMi made their first contribution in #381
• @johnnyleitrim made their first contribution in #377
• @menglewis made their first contribution in #385

Full Changelog: v0.8.1...v0.8.2

• Build against official Spark 3.0.0 release.
• Cross build: Spark 3.0/Scala 2.12, Spark 2.4/Scala 2.11, Spark 2.4/Scala 2.12 (Experimental).
• Fixed an error in shortest path (#367).
• Removed the use of deprecated methods (#363).

• Support Spark 2.4 / Scala 2.11 and Spark 3.0 (Preview) / Scala 2.12.

• Pregel API.
• Support Spark 2.4, Python 3.6, and removed support for Spark 2.2.
• Removed scala-logging-slf4j from dependencies.

We encourage all users to upgrade to this release because of motif finding bug fixes and fixes to the release process for the Python API!

Bug fixes

• [Correctness] Connected Components had a bug which affected correctness in some cases. It has been fixed.
• All previous versions of GraphFrames could encounter failures with motif finding using negated edges. These were not correctness issues, but they would cause Exceptions on perfectly valid queries.
• The last release was missing Python files required for the Python API. This release fixes that issue.

New features

• Filter vertices or edges to select subgraphs

Support for Apache Spark 2.2 and 2.3

Thanks to the many contributors to this release!

We strongly encourage all users to use this latest release because of the bug fix described below!

CRITICAL bug fix

• All previous versions of GraphFrames had a bug which can affect ConnectedComponents and other algorithms.
• This fixes a bug in indexing vertices with non-Integer IDs.
• It can affect all algorithms which are wrappers around GraphX, including ConnectedComponents, PageRank, and others.
• The bug surfaces when the input DataFrame is non-deterministic. E.g., running an algorithm on a DataFrame just loaded from disk should be fine in previous releases, but running that algorithm on a DataFrame produced using shuffling, unions, and other operators can cause incorrect results. This issue is fixed in this release.

New features

• Python API for aggregateMessages for building custom graph algorithms
• Scala API for parallel personalized PageRank, wrapping the GraphX implementation. This is only available when using GraphFrames with Spark 2.1+.

Support for Spark 1.6, 2.0, and 2.1

Contributors to this release

• Felix Cheung
• Tuomas Sivula
• Xiangrui Meng
• Joseph Bradley
• Bagrat Amirbekian
• Santiago Castro
• Philip Yang