Contributing

Dataform is a TypeScript project, using Bazel as a build tool. To scope out work, please check existing issues (which includes feature requests) or open a discussion thread!

Getting Started

First 🍴 fork this repository, clone it to your desktop, and navigate within.

Requirements

Bazel

Bazel is a build system which we used to build the project and run tests.

The easiest way to install the correct Bazel version is through Bazelisk via NPM:

npm i -g @bazel/bazelisk

Run the CLI

You can run the project as you would the npm installation of @dataform/cli, but replace dataform with ./scripts/run.

For example, to print out the default help information:

./scripts/run help

Check the docs for more examples.

Note: If you are running Bazel on a Mac, this or any step that requires building may fail with a Too many open files in system error. This is due to a limitation on the default maximum open file descriptors. You can increase the limit by running sudo sysctl -w kern.maxfiles=<LARGE_NUMBER> (we use 65536).

Test

The following command runs tests for @dataform/core:

bazel test //core/...

If you need to run integration tests, that rely on encrypted secrets, please get in touch with the team.

Lint

The following command to check for any linting errors

./scripts/lint

Building

Building the CLI will build most of the required components.

bazel build cli

The projects folder here is not built as it requires an environment file, which can be provided from the team.

Add New NPM Dependencies

Global yarn installations will throw errors when installing packages, instead you should use:

$ bazel run @nodejs//:yarn add ...

Additionally, installed NPM dependencies need to be added to the deps of ts_library rules by prefixing them with @npm//....

The Contribution Process

Decide on what you'd like to contribute. The majority of open-source contributions come from:
1. Someone deciding they want a feature that is not currently present, and which isn't a priority for the team.
2. Embracing the community aspect of open source (or getting that commit count up) and solving an issue.
Plan out the change, and whether it is feasible.
1. If you're unsure of the scope of the change, then ask in the issue or create a discussion.
2. We'd much prefer multiple smaller code changes than a single large one.
3. Avoid changing core functionality over a long time frame. Our development process is very dynamic, so if your code depends on lots of other parts of the project, then it is likely to be out of date by the time you finish!
4. If you're solving an issue, be sure to comment to make it known that you are currently solving it. Unless we have worked with you before, it is unlikely that we will lock the issue to you.
Begin materialising your masterpiece.
1. Create a feature branch based on main (link) for development work.
Once done, review your code, run the tests, check for common mistakes and then open a pull request.
1. Tidy the code by removing erronous log statements. Comment difficult to interpret sections. Make sure functions are names appropriately. We will review the pull request mainly by the git difference.
2. Assign a reviewer. Pick anyone on the team who seems to contribute a lot and they will refer it onto whoever is most responsible for the given subsystem.
Discuss and process any changes requested.
1. It's unlikely your pull request will be perfect immediately; there will likely be some changes requested, whether it's to do with style or a more fundamental issue.
2. The automated integration tests must pass.
3. Once a pull request is accepted and all automated integration tests are passing, we will merge it for you.

Reporting Issues (!)

Another way we'd love for you to contribute is by flagging any issues you find. First check through the list of existing issues for anything similar, in order to avoid duplicates. If not, then full steam ahead!

Promoting Dataform

If you're using Dataform for interesting projects then please let people know! Reach out to dataform-preview@google.com for marketing support.

Common Pull Request Mistakes

Is it too long? Small pull requests are easier to review and merge. If you are planning on making a larger change, then talk to the team and write a document on the design.
Have you appropriately increased test coverage? If the operation of the change is not already tested, then tests will need to be written.
Is it a hack? Does it solve the problem, but it is not reliable, reproducable or extendable? In other words, does it smell?
Have you changed whitespace or touched unrelated code? Please avoid this as it makes pull requests far more difficult to review.
Are the comments useful, and is the code readable? Are the function and variable names appropriate?

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!