How to make reviewing pull requests a better experience

Invest in teaching communication skills

When an Open Source project is first getting started, as a maintainer you’re probably excited to see any community interest in contributing. But as the community grows, it’s typical to feel swamped by incoming pull requests. Even if the volume is manageable, reviewing low-quality pull requests can feel like a drag. And that can lead to putting off community pull request (PR) reviews in favor of other priorities. Even promising work may sit unreviewed for weeks if maintainers aren’t able to review it all.

There is a better way, however. As Open Source maintainers, we ought to continually assess the code-contribution and review processes for our projects, and deliberately work to make them better. In this post, we focus on a specific area of investment that in our experience can pay huge dividends: teaching communication skills.

Zulip is an Open Source team chat application designed for seamless remote work. The Zulip project moves forward by routinely merging contributions to the core product from the community.

In the last six months, we’ve integrated code from 125 contributors (many of them college students) with a core team of just 13 engineers. How do we make it work?

We think of our project’s code contribution process as a product, with its own key personas, goals, and features (more on that in a separate post). That means spending time iterating on the contribution process so we can identify pain points and opportunities for improvement, just like any other product we’d bring to market. 

Teaching contributors excellent communication skills is a central feature of our contribution process that we’ve invested in over time. In this blog post, we’ll focus specifically on one key aspect of communication for Open Source projects: presenting a PR for review.

Intentionally teaching contributors this skill enables maintainers to make excellent use of their time in reviewing PRs for two reasons:

1. It takes much less time to review work that’s presented clearly.
2. Providing initial feedback on communication before diving into the code helps maintainers easily identify which PRs to invest time in.

We’ll dive into how we teach contributors to present their work well. We’ve found that the approach described here, after some initial time investment, makes the review process much more efficient for our project’s maintainers. It’s also a great learning experience for contributors, who gain confidence and develop the motivation to continue to contribute to Zulip and other Open Source projects.

Teach contributors how to present a PR well

What makes PRs from your strongest contributors stand out, even for minor changes? In our experience, it’s most often how clearly contributors communicate about their work, which makes a substantial difference in how easy a PR is to review.

Some aspects of communicating clearly will be specific to your project, while others are quite general. Regardless, clearly explaining the work contained in a pull request is a teachable skill. But you might see the problem: teaching communication from the ground up to each contributor doesn’t scale well. We’ve addressed the problem of scale using the following three-part approach to teaching communication:

1. Write detailed, precise documentation that explains what you’re looking to see in a pull request. In the long run, you want documentation that will get a new contributor who is careful and thorough 80%+ of the way to where you want them to be.
2. Provide clear, actionable feedback when some aspect of a contributor’s communication misses the mark.
3. Iteratively improve your documentation based on pain points you see contributors encounter and feedback you find yourself offering with some regularity.

Develop detailed, precise documentation

Documenting how to communicate on a pull request might sound challenging, but your experience of reviewing PRs should give you a very good idea of what you’re looking for from contributors. What makes a great pull request easy to review? Write down as many details as you can, and work them into a set of guidelines or instructions.

Don’t worry about perfecting your documentation at the outset. As we discuss below, writing great documentation is an iterative process.

For example, for Zulip, we consider a pull request to be a presentation of proposed changes. The contributor’s aim should be to explain their changes to reviewers as clearly as possible. We make it clear upfront that contributors will need to put in quite a bit of work to present their pull request, beyond writing the code.

Here are some best practices identified from the experience of documenting our expectations and observing how contributors respond to our PR guidelines:

1. You don’t need perfectly polished contributor docs, but make an effort to present information clearly and directly. Use numbered steps, bullets, headings, etc. to organize content. Cross-link related documentation.
2. Get as specific as you can about what makes for good, reviewable work. Don’t assume you’re being too picky or asking for too much, especially when your requirements represent quick tasks, like taking a good screenshot of a feature that requires manual testing. It’s much easier for contributors to follow specific instructions than vague ones. Link to standout examples of past PRs that were especially well done, or illustrate specific aspects of what you’re looking for.
3. Share some reasoning behind your requirements alongside the instructions. This will help folks follow them correctly, and provide additional motivation.
4. Make sure contributors see your docs! Consider linking to them from an overall guide about your contribution process and/or a pull request template in GitHub.

For example, we ask contributors to include the following in the description that accompanies their Zulip pull request. You may find that similar points are also helpful for your project:

• Provide an overview of your changes.
• Note any differences from prior plans (e.g., from the description of the issue you are solving).
• Call out any open questions, concerns, or decisions you are uncertain about. The review process will go a lot more smoothly if points of uncertainty are explicitly laid out.
• Include screenshots for all visual changes, so that they can be reviewed without running your code.

Provide clear, actionable feedback on communication

No matter how clear and detailed your documentation is, the communication on pull requests for your project will never be perfect. To help contributors improve, you should give direct feedback not only on the code changes but on the way their work is presented as well.

This might sound like more work, but with the right approach, you’ll likely find that reviewing PRs becomes less work. Imagine you get a poorly explained pull request from a new contributor. You’d feel bad about ignoring it, so you spend time wading through it and give extensive feedback on what needs to be improved. The contributor fixes a handful of typos you pointed out, and then bounces without making the bigger structural changes that were required. What a waste of time!

You could have saved time by focusing on communication first. You don’t need to immediately tell the contributor everything they’ll need to do to make their PR ready to merge. Instead, give them just a handful of next steps for getting their PR closer to being merged. This should be easier for the contributor than addressing a mountain of feedback all at once. Then, you can provide the next set of steps only if they successfully get through the first steps.

Starting your review with feedback on communication, and not feedback on the code, is often a better strategy because:

1. Generally, it’s easy for maintainers to spot communication gaps and recommend steps to address them. Is the PR missing a description, screenshots, or a link to the issue it’s solving? Are there typos in commit messages? If the PR doesn’t follow some of your documentation, a quick note linking to the relevant documentation is all you need to leave as an initial review. 
2. A contributor’s willingness to address your clear, direct communication feedback typically suggests that the rest of the review process will go much more smoothly.

In short, the initial stage of review should help you identify contributors who are likely to be successful in your project. If a contributor is unable or unwilling to communicate clearly enough to explain their work, even with explicit guidance from you, they are unlikely to succeed in making code changes that you’ll be able to confidently integrate.

A few tips about providing communication feedback:

1. As always, be professional and respectful, while also being direct and to the point. Using too many polite turns of phrase can make contributors feel unsure whether or not they really need to address your feedback, or make your feedback harder to understand.
   • Good: “Please answer all the questions in the pull request template to prepare your PR for review.”
   • Less good: “If you wouldn’t mind, it would be good if you could answer all the questions in the pull request template.”
2. Give just enough guidance for a thoughtful contributor to be able to address your feedback. Don’t spend a lot of time walking the contributor through what to do step-by-step when they should be able to figure it out using your documentation and other easily accessible resources.
   • Good: “Please add before/after screenshots to your PR description; see guidelines.”
   • Less good: “Your PR doesn’t have screenshots.” — This states a fact but doesn’t say what, if anything, should be done about it.
   • Less good: “Please add before/after screenshots to your PR description. Do you have screenshot software you like to use? Ping me on chat if you need help. Here are the 5 screenshots you should take:…” Providing this level of support is likely not a good use of the reviewer’s time, and will not be sustainable as you continue to work with this contributor. It also sets contributors up to be overly dependent on you, rather than figuring out that level of detail on their own.
3. Focus on whether the contributor got their message across clearly over details that don’t inhibit comprehension, e.g., occasional typos, or details that might be tricky for non-native speakers of your language to get right. Your goal is to teach the contributor to communicate effectively with reviewers, so that’s the bar you are aiming for.
   • Good: “You need to explain this part in more detail.”
   • Less good: “You wrote ’posibly’ instead of ‘possibly’ in the second sentence.”

Iteratively improve your documentation

As you provide communication feedback on pull requests, you’ll notice patterns in the types of feedback you are giving. Whenever you’re giving the same guidance again and again, it’s probably time to update your documentation to give contributors that guidance upfront. If it feels like a side point, drop it into a “more tips” section at the bottom. Over time, you may start splitting out major sections into dedicated pages you can link to from elsewhere.

As your documentation gets longer and more detailed, you should expect that contributors won’t necessarily read all of it upfront. That’s OK! Your documentation will still increase in value to you: rather than taking the time to explain a problem each time it appears, you’ll simply link to the best version of the explanation that you wrote earlier. For example, on a PR that’s missing a description, you’ll be able to write: “To prepare your PR for review, please add a description following these guidelines.” By saving this reply, and others like it, in GitHub, you’ll be able to guide contributors on steps they need to take in just a few seconds.

What’s next?

The work of refining your contributor documentation and your feedback process is never finished. Having worked with hundreds of Zulip contributors over the years, we’re still making improvements, including overhauling major sections of the docs every year or so. As we make these changes, the quality of PRs from new contributors improves over time. Providing feedback becomes easier as well, with better resources to point to.

In future posts, we expect to go into other aspects of developing effective ways to work with contributors, including teaching other communication skills (e.g., commit structure, responding to review feedback), setting up an efficient multi-step review process and more. Subscribe to the Zulip blog to stay in touch!

This article is part of our series on Practical Open Source (POSI). POSI
aims to facilitate discussions about doing business with and for Open
Source.