Of all CI systems I’ve used during my software development career, both as developer and manager, GitLab CI has proven itself to be the overall best system out there.

First of all, for a CI system to fulfill its purpose, it needs to test every git commit to validate that no code change breaks the test suite no matter how small the change is. Having code hosting and CI integrated in the same system is the obvious way it should be, and GitLab CI does that integration well on all levels from git command-line option support to repository permission control to user interface in every view.

For developers to pay attention to tests that stop passing after their code changes, there need to be automatic and easy to read notifications. GitLab CI does email and dozens of integrations, such as Slack messages.

For developers to quickly find the error, root cause it and deliver a proper fix, the CI pipeline needs to be visually clean yet offer options to drill into logs and build artifacts. GitLab CI checks all the boxes here.

Finally, when it is time to review a code change, the way GitLab CI integrates with GitLab Merge Requests is seamless. For example, a human reviewer can, after reading the code change, choose the action Merge automatically if CI pipeline passed without having to attend the pipeline. Brilliant time-saver.

GitLab is also open source, so if it is missing a feature that is critical for your software development work, you can add that feature yourself (or pay somebody to do it). Being open source also brings many other benefits and ultimately sets it apart from GitHub, its traditional closed source rival.

Last but not least, GitLab has excellent documentation anybody can dive into easily. Therefore, instead of duplicating that by explaining the general GitLab features and benefits, I will instead showcase how it is used in a real-life project: MariaDB Debian package maintenance.

# Salsa - Debian’s GitLab instance

Debian launched salsa.debian.org in early 2018 as a platform for Debian developers to host the source code of Debian packages. In July of 2018, Salsa-CI, which provides a standardized GitLab CI pipeline template for Debian packaging quality assurance, was launched. I adopted this in August 2018 for all the packages I maintain in Debian, of which by far the biggest is the MariaDB database server.

Over the years it has grown to a very extensive pipeline that, in addition to the inherited general Salsa-CI steps, also runs a variety of additional test jobs, including:

• Building MariaDB in parallel on multiple Debian releases and processor architectures
• Building consumers of the MariaDB Client C library to ensure the interface stays stable
• Upgrading old versions of MariaDB to the latest one, both full server upgrades and partial small upgrades, such as the client library upgrades
• Upgrading various versions of MySQL, Percona and others to ensure that cross-upgrades from MariaDB variants work
• Upgrading various combinations of Debian releases and MariaDB, simulating full system upgrades
• Running static analysis to detect security issues and general software quality issues

Not only does the pipeline do all this, but it is also optimized to use ccache and other techniques to run as fast as possible. For details on exactly what the pipeline does, one can simply read the file debian/salsa-ci.yml. Normal GitLab CI uses the file .gitlab-ci.yml in the project root, but since in Debian packaging one is only allowed to modify files under the debian sub-directory, the file resides at this customized path.

The fact that the structure and all the steps the CI runs are defined in the code repository itself is very powerful. Anybody inspecting a pipeline run of a specific git commit can always find the specific version of the GitLab CI definition in the source code of that exact git commit itself. This is vastly superior to the structure e.g. Buildbot or Jenkins uses, where the pipeline is defined separately from the code it tests.

Not only does this make it much easier to read the pipeline steps, but it also makes contributing to the pipeline code as straightforward as filing a Merge Request on the repository, just like with any other file. The fact that the CI code and actual software code are together in the same repository makes it much easier to enforce a rule that CI must always pass, as any commit that changes the software behavior can at the same time also update the CI pipeline to account for that intentional change in behavior. Needless to say, GitLab CI works seamlessly with the protected branches feature and Merge Requests in GitLab to ensure easy and sensible rules to enforce that the mainline always stays green.

Using standard GitLab CI features, there is also a scheduled monthly rebuild that reveals if an update in any of the dependencies causes the pipeline to fail in the absence of new code commits.

# Case example: MariaDB 10.6.9 incompatibility with Percona XtraDB 5.7 temporary table format

To illustrate GitLab CI in action, let’s take a look at a recent example in which it prevented breaking upgrades for (some) Debian users. In June 2022, the Salsa-CI pipeline for MariaDB 10.6.8 was all green and upgrades from Percona XtraDB 5.7 were passing flawlessly. However, after importing a new upstream minor maintenance release on August 16th, the Salsa-CI pipeline for MariaDB 10.6.9 started failing on the Percona upgrade. From the CI job log, it was easy to see that MariaDB failed to start with the /var/lib/mysql data directory from Percona XtraDB 5.7. From the build artifacts, it was easy to inspect the precise error message from the MariaDB server (as build artifacts are configured to expire after 30 days, I can’t link to them for reference).

This quickly led to filing MDEV-29321 on August 18th. As the failure was caught by CI, it was easy to provide the steps to reproduce in the bug report, along with logs and CI job references. This helped the upstream developer to immediately pinpoint the issue and post a patch to fix it, which was validated by a Salsa-CI test run on a temporary development branch. The fix was applied on mainline in the Debian packaging repository of MariaDB on August 24nd.

Such a quick turnaround time would not have been possible without a good CI system. The process clearly benefited from the very clear user interface of GitLab CI that made it easy for all parties – even those who didn’t have prior experience of GitLab CI – to read the pipeline and inspect the logs.

# Computers are good at repeating the same tasks over and over; humans are good at exploring visual things

Through the years, Salsa-CI (GitLab CI) has proven incredibly valuable – it has been able to catch the tiniest packaging mistake immediately as the commit is pushed to Salsa (GitLab) and the git mainline stays in a condition that can be shipped at any time. This makes it possible to import new upstream releases at any given time, and to upload them to Debian with a high confidence that nothing will break. Before Salsa-CI, the MySQL and MariaDB packages had hundreds of open bugs in Debian (and also Ubuntu, which inherits the packages from Debian). Now genuine new bugs are rare, staying consistently in the lower tens.

If you are developing software professionally but not using CI in all your projects, you should definitely start now. Computers excel at running repetitive tasks over and over, and CI is exactly that. No human would have the diligence to always test everything. Humans tend to test code changes only when they have doubts, and thus most bugs slip in when a human makes a small change they don’t think can break anything – and then it breaks. These cases always come as a surprise, and you don’t want to find it out in production use but rather delegate it to a CI system to validate everything.

What humans are good at is visual inspection. A large part of our brain cortex is devoted to vision, so we should leverage it. GitLab CI does a great job converting the repetitive CI test run results into pipelines with various colors and symbols, perfect for human consumption. Humans also have an eye for beauty and elegance, and I personally enjoy exploring the GitLab CI pipelines. If you haven’t already, please try it out, and enjoy the warm fuzzy feeling of seeing all green pipelines!

# So easy and intuitive you can start using it without reading lengthy manuals

This blog was just a sneak peak into what GitLab CI can do – there are so many useful features, such as scheduled pipeline runs (to detect regressions introduced from updated dependencies), automatic repository mirroring, and many UI goodies, such as badges. I recommend skimming through the GitLab CI documentation and in particular the .gitlab-ci.yml reference and then just start playing around with it. GitLab.com has a free plan for basic use which (unlike GitHub.com)) also includes private repositories.

In short, GitLab CI makes the life of a programmer (or a team of programmers) significantly more productive by ensuring that tests run all the time, failures are quick to detect, and the whole test system is handy to maintain and evolve together with the code. Using it feels like a breeze, both because the features work just like one would expect and because the user interface is very easy to navigate. There is rarely a need to read documentation to get started – just try GitLab CI today and discover its power yourself!

Want to read more? See also the GitLab.com blog post “Debian customizes CI tooling with GitLab”