Contributing Guide

How to Contribute to this Project

1. Clone this repository:

git clone https://github.com/pycontw/pycon-etl

2. Create a new branch:

Please checkout your branch from the latest master branch before doing any code change.

# Checkout to the master branch
git checkout master

# Ensure that's you're on the latest master branch
git pull origin master

# Create a new branch
git checkout -b <branch-name>

3. Make your changes.

If your task uses an external service, add the connection and variable in the Airflow UI.

4. Test your changes in your local environment:

Ensure that the dag files are loaded successfully.
Verify that the tasks run without errors.
Confirm that your code is properly formatted and linted. See Convention section for more details.
Check that all necessary dependencies are included in the pyproject.toml file.
- Airflow dependencies are managed by uv.
Ensure that all required documentation is provided.

5. Push your branch:

git push origin <branch-name>

6. Create a Pull Request (PR).

If additional steps are required after merging and deploying (e.g., add new connections or variables), please list them in the PR description.

7. Wait for the review and merge.

Convention

Airflow Dags

Please refer to 「大數據之路：阿里巴巴大數據實戰」讀書心得 for naming guidelines.
Table name convention:

Code Formatting

Please run make format to ensure your code is properly formatted before committing; otherwise, the CI will fail.

Commit Message

It is recommended to use Commitizen.

Release Management (CI/CD)

We use Python CI and Docker Image CI to ensure our code quality meets specific standards and that Docker images can be published automatically.

When a pull request is created, Python CI checks whether the code quality is satisfactory. At the same time, we build a cache image using Dockerfile and a test image with Dockerfile.test, which are then pushed to the GCP Artifact Registry.

After a pull request is merged into the master branch, the two image tags mentioned above are created, along with a new staging tag for the image generated from Dockerfile.

Once we verify that the staging image functions correctly, we merge the master branch into the prod branch through the following commands.

git checkout prod
git pull origin prod

git merge origin/master

git pull origin prod

This triggers the Docker Image CI again to update the cache, test, and staging images, as well as to create a latest image that we will later use for deploying to our production instance. See the Deployment Guide for the following steps.

---
config:
  theme: 'base'
  gitGraph:
    mainBranchName: 'prod'
    tagLabelFontSize: '25px'
    branchLabelFontSize: '20px'
---
      gitGraph
        commit id:"latest features" tag:"latest"
        branch master
        commit id:"staging features" tag:"staging"
        checkout prod
        commit id:"prod config"
        checkout master
        branch feature-1
        commit id: "new features" tag:"cache" tag:"test"

Loading

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!