HACKER Q&A
📣 blunte

Why is daily or n * daily deployment so popular?


Another job listing has appeared which promotes multiple daily deploys as a positive bullet point.

Is this really beneficial? Has there been any meaningful analysis of the use of CI/CD - in parts or in whole?

My limited experience with CI/CD has shown significant overhead, both human and compute, but much less end user value. Aside from research, I would enjoy hearing both positive and negative experiences people here have had.


  👤 matthewmacleod Accepted Answer ✓
Frequent deployments emerge as a side effect of a good CI/CD process, rather than themselves being the thing that prompts it.

They can be taken as a signal that the deployment process is safe, automated, and low-overhead. When that applies, I have generally seen the number of deployments go up. The reason is pretty simple: engineers stop worrying about deployments, the overhead goes away, and deployment becomes part of the "definition of done".

My experience is that infrequent integration and deployment is a more painful process when it comes to integration and deployment time. Easy deployments encourage features and bug fixes to be rolled out as atomic units when they are completed, rather than saving them for big-bang releases. This encourages small, independent changes, rather than larger ones that involve more simultaneous code and data model changes – often in my experience multiplying the chance of failure by an outsized factor.

Honestly I'd actually go in entirely the opposite direction and say that one of the most valuable and beneficial systems I see in companies is an effective CI/CD system with frequent deploys. I find it discourages bad practices and encourages good ones; I would consider it a red flag when looking to work with a company if they didn't have such a process in place.


👤 richbradshaw
Yes.

Deployments should be quick, simple, reliable and accessible to developers. With automated builds + unit tests you can have simple deployments that take < 10 minutes if not < 5 minutes without too much complexity on most medium sized projects.

This lets you ship often and break complex behaviour into smaller tasks. Exposure to bugs is limited to only what's shipped, so you can be confident that each deployment is correct.

I think the idea that there is significant overhead seems to me to be the opposite - frequent small deployments are simple and easy to verify. Single large deployments often change huge amounts of code, multiple DB migrations, and introduce multiple opportunities to fail.

If you deploy too infrequently, you can also develop a culture of rushing to get into the release, or being (or your customers being) scared to push to production due to bad experiences in the past.


👤 bjourne
I've been working on a project that integrated/deployed so rarely that we had a dedicated merge team consisting of about half the work force. Except the merge team wasn't involved in the actual development process so when they were trying to merge they had no idea how the various pieces should fit together. So the development team had to guide them trough the process. It was painful and it frequently took months for new development to reach the testing infrastructure and then many more months before reaching the customers. Debugging crashes on customer software meant checking out releases a year old or older and trying to figure out what could have been the root cause.

👤 thedevindevops
The Release early, release often philosophy promotes a tighter feedback loop, allowing the catching of problems early and (with the framework of unit and integration tests supporting it) measurable increases in code quality.

👤 giantg2
I think it depends on the policies set by each company as to whether this practice is helpful or hurtful. If done right, CI/CD is great.

My company has embraced CI/CD. Unfortunately, they have created metrics around it. In some instances this has created headaches because managers want us to do stuff to make the metrics look good. I've seen the same thing with other practices in our company, such as the story points in Agile.


👤 borplk
Frequent deployment is almost always a reliable symptom of a mature team and pipeline.

When they list it as positive bullet point they are saying you are not going to be chasing "Jimmy the only guy that can do deploys" to get your work deployed and so on.

There are lots of crappy teams that deploy lots of times a day too though so keep that in mind as well.


👤 UK-Al05
The book accelerate has lots of statistics showing that companies who release frequently perform better

👤 Leherenn
One place where it does not work is when the user has to do something to update, e.g. flash a device.

In that case, you quickly get update fatigue from the user. From experience, every couple of months or so is a good trade-off.