This project houses the experimental client for Spark Connect for Apache Spark written in Golang.

Currently, the Spark Connect client for Golang is highly experimental and should not be used in any production setting. In addition, the PMC of the Apache Spark project reserves the right to withdraw and abandon the development of this project if it is not sustainable.

This section explains how to run Spark Connect Go locally.

Step 1: Install Golang: https://go.dev/doc/install.

Step 2: Ensure you have installed buf CLI installed, more info here

Step 3: Run the following commands to setup the Spark Connect client.

Building with Spark in case you need to re-generate the source files from the proto sources.

git clone https://github.com/apache/spark-connect-go.git
git submodule update --init --recursive
make gen && make test

Building without Spark

git clone https://github.com/apache/spark-connect-go.git
make && make test

Step 4: Setup the Spark Driver on localhost.

1. Download Spark distribution (4.0.0+), unzip the package.

2. Start the Spark Connect server with the following command (make sure to use a package version that matches your Spark distribution):

sbin/start-connect-server.sh

Step 5: Run the example Go application.

go run cmd/spark-connect-example-spark-session/main.go

To run the Spark Connect Go application in a Spark Cluster, you need to build the Go application and submit it to the Spark Cluster. You can find a more detailed example runner and wrapper script in the java directory.

See the guide here: Sample Spark-Submit Wrapper.

See Quick Start Guide

The overall goal of the design is to find a good balance of principle of the least surprise for develoeprs that are familiar with the APIs of Apache Spark and idiomatic Go usage. The high-level structure of the packages follows roughly the PySpark giudance but with Go idioms.

Please review the Contribution to Spark guide for information on how to get started contributing to the project.