Automate Pull Requests merging with Bors

Clémentine -

Meilisearch currently maintains more than 50 open-source projects on GitHub. In the last six months alone we merged over 1000 Pull Requests (PRs), 160 of which came from external contributors.

Ensuring the codebase remains stable and that code quality is always top-notch requires us to be very careful when it comes to merging a new PR. As the number of contributions grows, the automation of this process becomes a necessity in order to keep everything simple for the contributors and safe for Meilisearch's codebase.

In this post I will describe why we decided to use Bors, a merge bot for Github pull requests, to create a better environment for both maintainers and contributors.

Our merging process

During an open-source contribution, any changes must be done on their own separate branch. Afterwards, when the work on this branch is finished, one of our main requirements before merging is that the new branch is up-to-date with the primary branch—called main in most of our repositories.

If the PR is not brought up to date with main, you could say that most of the work is done, but most of the work is not enough for us. Even if all the automated tests pass on the PR itself, they are not guaranteed to pass when merged if the PR branch and main are not aligned.

Before using Bors, this meant we often had to ask PR authors to rebase their branch if, for instance, a different PR had been merged meanwhile. This made contribution more complex than we wanted, since not everyone is comfortable with Git’s rebase command.

Why rebasing?

At Meilisearch, we believe keeping a clear Git history is really important for two main reasons:

  1. It allows everyone to follow all changes made to the project
  2. When debugging, a clear history helps identify which commit introduced the issue

The goal of a rebase is to move the commits from a PR branch to right after the last commit on main. Therefore, rebasing keeps your Git history clear and consistent with the order in which you have merged your PRs.

Rebasing is not the only way to bring a branch up-to-date with the main branch: you can also pull the changes from main to the PR branch, for instance. The issue with this approach is that the pull would add a merge commit to the PR branch, which would not keep the repository history clear.

Keeping in mind how important it is to maintain a clear commit tree, rebasing is the only way to guarantee that the work that is going to be merged fits the current work in production. If the tests pass on an up-to-date branch, they will also pass once merged into main, ensuring your production branch will not break.

Finding the perfect tool

With our rebase requirements in mind, I began my search for the perfect tool. I wanted to be efficient, so I narrowed my scope and only evaluated tools that were:

  • Free and open source: the tool had to fit the values of Meilisearch and its community
  • Highly rated: the tool had to be stable and reliable, since any mistakes when handling PRs are bound to have impactful consequences
  • Well documented and easy to set up: good documentation saves time that we can use improving Meilisearch

Unfortunately, I did not find as many solutions as I expected within this scope.

The first tool I tried was Shipits, which I discovered thanks to this article on Shopify’s engineering blog. Though Shipits seemed really solid and promising, I felt it was a better fit for complex integration processes instead of open-source projects just trying to keep their Git history clean.

Another tool I tested was Kodiak. The set up was really easy, in large part due to their great documentation. It was a good candidate for our use-case, but the automatic merge process is based on labels that we would have to manage, which does not really fit our workflow. For example, our minimum requirement when merging a PR is obtaining the approval of one reviewer; sometimes, though, I want two people reviewing one PR and it is easy to forget to communicate this to Kodiak. The moment it receives the first approval, Kodiak automatically merges the PR because the one-reviewer-requirement has been respected.

Another thing we noticed was that Kodiak does not update the branch originating the PR by rebasing it and instead creates a Git commit by pulling the main branch into the PR branch, which, as we saw, muddles the repository’s history. In the end, Kodiak did not fit our needs, even if its setup and configuration provided a seamless and hassle-free experience!

Contrary to all the other tools I looked into, Bors was built following a unique and powerful principle described by the original Bors creator in this article:

The Not Rocket Science Rule Of Software Engineering: automatically maintain a repository of code that always passes all the tests.

This "rule" perfectly fit our requirements and made Bors our final choice.

How does Bors work?

Bors is a merge bot—meaning, it is an application that automates aspects of the PR merging process. Its setup is straightforward and, like Kodiak, very well documented. Indeed, configuration can be a single bors.toml file containing the names of tests we want to execute before merging.

All Bors commands have to be given through comments in a PR. To merge a PR, all we need to do is use bors merge.

Bors first applies a rebase before running the tests and, if they succeed, merges them back into main. To be more accurate, Bors does not actually rebase the branch of the PR: it merges the commits of the PR into another branch, called staging, that is already up-to-date with main. If the tests do not pass on staging, Bors returns a failure and does not merge the PR.

The icing on the cake: Bors is also able to manage merging multiple PRs at the same time, which means we don't need to merge them one by one. Each PR is added to a queue and will be checked with the last version on main at regular intervals.

If Bors finds conflicts between the PR and the main branch, it cannot make the decision about which final changes to apply. In this situation, the contributors have to rebase the branch manually—if that happens to you, I wrote this quick tutorial on how to rebase from a forked repository.

Conclusion: which problems have we solved?

We at Meilisearch have used Bors for a few months now and are really satisfied. We know that keeping the Git history clear while providing a stable codebase can be very complex, but so far Bors has helped us greatly optimize our workflow. With Bors integrated in our workflow there is:

  • No need to ask and wait for PR authors to rebase
  • No need to test and merge PRs one by one
  • No frustration for our contributors: they all do amazing work, but some of them are not really comfortable with Git’s rebase feature, which is indeed not always easy to use—especially when working from a forked repository!

Bors makes both maintainers and contributors happier; we can, therefore, recommend it for projects with the same use-case as ours.