After discussing the Antipatterns in the previous blog post, lets focus on getting things right in this one. There are many best practices and patterns when it comes to CI/CD - I tried to choose patterns that are either very important or often done wrong. The latter is the case for our first topic.
I found the next quote somewhere on the internet and I think it's spot on:
Here is some bad news:
With this in mind, Continuous Integration is a practice, not a tool. There is this really good blog post from 2005 which is still relevant and it underlines the message perfectly: http://www.jamesshore.com/Blog/Continuous-Integration-on-a-Dollar-a-Day.html.
If you want to do CI right, the developers need to understand that CI is all about integrating and testing code regularly. There are a few best practices that can help to start:
We'll cover the deployment pipeline in greater detail later on - but for now it is important to understand what it does and what it is good for. Here is a suitable definition from devops.com:
The (deployment) pipeline breaks down the software delivery process into stages. Each stage is aimed at verifying the quality of new features from a different angle, to validate the new functionality and to prevent errors from affecting your users.
Here is a chart showing a basic deployment pipeline, which can be a good starting point for medium complex projects:
As you can see, a software package will pass through multiple stages before it arrives in production. Each stage has a particular reason, which we will discuss in one of my future blog posts. Without discussing in detail what happens at the stages, there are two important things to know and to consider:
There is a direct relation between the level of automation (or the amount of manual tasks) and business success. It is astonishing. The Puppet DevOps report describes the relation in more detail here: https://puppet.com/resources/whitepaper/state-of-devops-report. Companies that are high-performers when it comes to automation will twice as likely reach their goals than average performers or low performers. High performers automate more than 75% of all build and deployment tasks and reduce the number of manual tasks significantly.
There is a basic rule for automating tasks in the IT: When you have to do it twice, start thinking about automating it. When you need to do it a third time, automate it.
Automation should be introduced at least in these areas:
The difficulties when implementing automation is that it always competes with implementing new features. Therefore it is not advisable to spent too much time at once to automate manual tasks since it will reduce the team capacity to implement features with business value. It is better to continuously increase the level of automation, starting with the most important. So usually we try to add CI/CD tasks or general automation tasks to each sprint.
This is an obvious thing but it is also the one rule that gets ignored far too often. The most important thing when it comes to CI/CD is that it's always in a working state. So whenever a part of the process fails, someone (the one that produced the failure would be best) needs to take care of it immediately. When your check-in breaks the build, fix it or revert it. If test execution fails, immediately review the failure and fix either the test or the tested code.
Try to run all your CI/CD steps as often as possible (multiple times a day would be best). First, if you do something often and regularly, it becomes easier. Second, if you integrate on a daily base, the changes that might cause errors become small and easy to understand which makes it easier to fix issues. On the other hand, if you wait too long and check-in a huge amount of code at once, it will become very hard to locate issues if errors arise.
You can't control what you can't measure - Tom DeMarcos famous quote is true for your delivery process as it is true for everything else. If you start to introduce CI/CD to your delivery process, how do you know if it is successful and whether it is a good investment?
So before you start, define some KPIs, measure them and define target values for them as a result of the improvements due to CI/CD. Doing so gives you the possiblity to measure your success and to prove the importance of your work to the senior management. Let me give you an quick overview of the most important KPIs for Software Development (we'll skip Quality Metrics here since it is a huge topic on its own):
Lead and Cycle Time are the most imporant KPIs when it comes to measure the delivery process - it is all about how long it takes to deliver a feature to the customers. Here is the definition we use (there are many different available...):
Lead Time: The Lead time clock starts when the request is made and ends at delivery.
Cycle Time: The Cycle time clock starts when work begins on a feature request and ends when the item is ready for delivery.
When it comes to CI/CD only, the cycle time is pretty interesting. Also if your company has already established some kind of Delivery Pipeline, it can be used to measure the cycle time pretty good. Aside from that, tools like Jira can help to measure the time it takes for a story from the first definition to release in production. Most likely you already have this information hidden somewhere in the tools you use.
A classic agile metric - the velocity of a team. Let's start with the obvious: you cannot compare the velocity of one team with another. It really doesn't work so don't try it. But what is possible is to measure the trend of a teams velocity. At first, while introducing CI/CD, it will have a negative impact on the velocity since developers have to work on automating manual tasks. But in total, CI/CD will increase the speed of your team significantly and Velocity is a nice KPI to show this increase.
When introducing CI/CD, these metrics are important to measure the success of the initiative itself. Because CI/CD will have an immediate and big impact on build and deployment related metrics. So measure them when you start, set a goal and try to move towards it step-by-step. Interesting KPIs are:
Certainly, there is much more to discuss here, for instance whether it is possible to save money by introducing CI/CD. I might write a separate blog post about Software Delivery KPIs and Metrics, but for now, if you are interested to talk more about this topic, just let me know.
About the author:
In his role as CSA, Walter is responsible for designing and implementing fitting solutions for well-known enterprise companies. As a huge fan of Continuous Improvement, Walter is currently focusing his research on DevOps, CI/CD and agile methods.