What are your CI timings?

Question

A year ago, I started working in a mid-size (100 devs, single monorepo, bazel, 50 microservices) company with a relatively slow CI process - 30 minutes from commit to a green check. Both ICs and the managers in the organization was ok with it, I was pretty shocked, 20 minutes for feedback? I want it to be more like 5 minutes. During the last year, I started to measure, debug, and speedup the CI process (while waiting for the builds) and was able to reduce it to 6 minutes in the 50th percentile, 15 minutes in the 95th percentile. While After this project I've got appreciated by my colleagues, but I still feel it's not enough. Maybe I'm asking too much from our CI process? How much is it takes from a commit until you get feedback from the CI pipeline? How many devs?

leipert · Accepted Answer

I would say it heavily depends on what you do in your CI pipelines, if you only run linters and unit tests, 6 minutes sounds fine. If you run Integration test or have heavy compile times (Scala, Rust?) half an hour to an hour sounds reasonable to.
GitLab’s pipelines run between a few minutes and an hour. We execute pipelines based on what changed. If only docs changed: few minutes, if Frontend or Backend changed, it can take considerably longer (up two an hour).
Please note that this includes heavily parallelized Integration (and E2E) test. This is generally fine because I make sure that linters and tests pass locally for the things I changed before I push.
Also as code review takes quite a while, at least 6 eye principle, colleagues in different time zones, Pipeline execution of time is just something of a minor importance to consider.
I am unsure how many Engineers we currently have exactly, but I’d guess around 450?

sethammons · Answer

We have multiple services and a staging environment. Each (most) of the services can be auto promoted to stage via their build pipeline and then we have different whole system, black box tests for high level feature and regression tests.
Most services can pass their own ci between 30s and 5min. Then the system level staging tests can be 5-20min. Multiple teams deploy multiple times per day.
For our team, from merging to production deployment can take from 5min to 30min. Each service has unit tests that run fast, seconds at most. Then they have acceptance tests that spin up all or most dependencies in docker compose and blackbox test the service.

cottonseed · Answer

I think it's going to largely depend on build times and how much infrastructure you need to spin up during during tests.
Some discussion for the same question in a recent thread: https://news.ycombinator.com/item?id=21679714
We're quite a bit smaller but have similar numbers: 15-20m right now. We're dominated by build time (build caching might help) and schlepping docker images.

aprdm · Answer

Depends in the code base... a big C++ project can take a long time to compile and tests even in beefy machines. Some of our biggest projects are in the 3-6h mark using ccache.