《Continuous Delivery》 Notes 1: The problem of delivering software

What is "Deployment pipeline"?

A deployment pipeline is an automated implementation of your application's build,deploy,test and release process.

Typically a deployment pipeline includes:

Commit stage(Compile/Unit test/Analysis/Build installers)
Automated acceptance testing
Automated capacity testing
Manual testing(Showcases/Exploratory testing)
Release

The aim of the deployment pipeline:

it makes every part of the process of building, deploying, testing and releasing software visible to everybody involved, aiding collaboration.
it improves feedback so that problems are identified and so resolved as early as possible.
it enables teams to deploy and release any version of their software to any environments at will through a fully automated process.

Some Release Antipatterns

Deploying Software Manually

The evidence includes:

The production of extensive, detailed documentation that describes the steps to be taken and the ways in which the steps may go wrong.
Reliance on manual testing to confirm that the application is running correctly.
Frequent calls to the development team to explain why a deployment is going wrong on a release day.
Frequent corrections to the release process during the course of a release.
Environments in a cluster that differ in their configuration.
Releases that take more than a few minutes to perform.
Releases that are unpredictable in their outcome, that often have to be rolled back or run into unforeseen problems.
Sitting bleary-eyed in front of a monitor at 2 AM the day after the release day, trying to figure out how to make it work.

Why we need automated process?

Error will occur every time if the process is not fully automated.
It will need much more time to debug the environment and investigate any issues.
If we do not use automated process, we usually need to create a document to describe how to do the job, but the document is generally incomplete or out of date at any given time.
Automated process encourage collaboration.
Performing manual deployment is boring and repetitive and yet needs significant degree of experience.
We do not have any way to test a manual deployment process.

The automated deployment process must be used by everybody and it should be the only way in which the software is ever deployed.

Deploying to a Production-like Environment Only after Development Is Complete

It looks like:

If testers have been involved in the process up to this point, they have tested the system on development machines.
Releasing into staging is the first time that operations people interact with the new release.
Either a production-like environment is expensive enough that access to it is strictly controlled, or it is not in place on time , or nobody bothered to create one.
The development team assembles the correct installers, configuration files, database migrations, and deployment documentation to pass to the people who perform the actual deployment.
There is little collaboration between the development team the the people who actually perform deployments to create this collateral.

We need to integrate the testing, deployment and release activities into the development process.

Manual Configuration Management of Production Environment

It looks like:

Having deployed successfully many times to staging, the deployment into production fails.
Different members of a cluster behave differently.
The operations team takes a long time to prepare an environment for a release.
You can not step back to an earlier configuration of your system.
Servers in clusters have different versions of operating systems, third-party infrastructure, libraries or patch levels.
Configuration of the system is carried out by modifying the configuration directly on production environment.

All aspects of each of your testing, staging and production environment should be applied from version control through an automated process.

How can we implement deployment pipeline?

Regarding how to deliver software to our client, both speed and quality is important, and we need to consider the following principles

Automated, which means you can run build, deploy, test and release in a short cycle and the process is repeatable.
Frequent, which means you can run the whole process as many times as you can to make sure the quality is as expected.

There are some best practice we should take into consideration.

Every Change Should Trigger the Feedback Process

We can decompose any software application into 4 components:1) executable code; 2) configuration; 3) host environment; 4) data.

As to feedback process, we need to consider:

The process of creating the executable code must work.
The software's unit tests must pass.
The software should fulfill certain quality criteria.
The software's functional acceptance tests must pass.
The software's nonfunctional tests must pass.
The software must go through exploratory testing and a demonstration to the customer and a selection of users.

The feedback Must Be Received as soon as possible

So the deployment pipeline requires:

Run fast.
They are as comprehensive as possible
If any of them fails, it means our application has a critical fault and should not be released under any circumstances.
They are as envrionment-neutral as possible.

The Delivery team Must Receive the Feedback and then Act on it

It is essential that every body involved in the process is involved in feedback. We need to broadcast the feedback to any one who need it. and the feedback is no value unless we do not handle it.

What is the benefit?

Empowering Teams

Testers can select older versions of an application to verify changes in behavior in newer versions.
Support staff can deploy a released version of the application into an environment to reproduce a defect.
Operations staff can select a known good build to deploy to production as part of a disaster recovery exercise.
Releases can be performed at the push of a button.

Reducing Errors

Errors can creep into software from all sorts of places.

The cost of fixing a defect is different at different stages of the project. we need to fix the defect as early as possible.

With deployment pipeline, we can do it.

Lowering Stress

We do not need to face a battle when we try to release a new version.

Deployment Flexibility

We need to user automated deployment process to prepare a new environment for deployment and deploy the chosen version of the application.

Practice makes Perfect

With deployment pipeline, we can try as much as possible to deploy and manage our application in the same way.

With deployment pipeline, we do not need to spend time and money on lengthy and manually intensive testing at the end of the project.

Every Check-ins Leads to a Potential Release

This means that we need to make sure each check-in we make does not destroy any existing craft.

In Software, when something is painful, the way to reduce the pain is to do it more frequently.

If your tests are sufficiently comprehensive and you are running tests on a sufficiently production-like environment, then the software is in fact always in a releasable state.

Principles of Software Delivery

Create a Repeatable, Reliable Process for releasing Software

We need to automate almost everything, and keep everything you need to build, deploy, test and release your application in version control.

Automate Almost Everything

Please note that automation is a prerequisite for the deployment pipeline, as it is only through automation that we can guarantee that people will get what they need at the push of a button.

Keep Everything in Version Control

It includes requirement document, test scripts, automated test cases, network configuration scripts, deployment scripts, database creation, upgrade, downgrade and initialization scripts, application stack configuration scripts, libraries, toolchains, technical documentation and so on.

If It Hurts, Do it More Frequently, and Bring the Pain Forward.

Build Quality In

This is about continuous improvement, Everybody in the delivery team is responsible for the quality of the application all the time.

A fire alarm is useless if everybody ignores it. Delivery teams must be discipline about fixing defects as soon as they are found.

Done Means Released

It is not in the power of one person to get something done, it requires a number of people on a delivery team to work together to get anything done.

Everybody Is Responsible for the Delivery Process

Starting by getting everybody involved in the delivery process together from the start of a new project and ensure that they have an opportunity to communicate on a frequent regular basis.

Continuous Improvement

The Deming Cycle: Plan, Do, Study and Act.

Life is too short to spend our weekends in server rooms deploying applications.