How Google TestsSoftware - Crawl, walk, run.

One of the key ways Google achievesgood results with fewer testers than many companies is that we rarely attemptto ship a large set of features at once. In fact, the exact opposite is oftenthe goal: build the core of a product and release it the moment it is useful toas large a crowd as feasible, then get their feedback and iterate. This is whatwe did with Gmail, a product that kept its beta tag for four years. That tagwas our warning to users that it was still being perfected. We removed the betatag only when we reached our goal of 99.99% uptime for a real user’s emaildata. Obviously, quality is a work in progress!

It’s not as cowboy a processas I make it out to be. In fact, in order to make it to what we call the betachannel release, a product must go through a number of other channels and proveits worth. For Chrome, a product I spent my first two years at Google workingon, multiple channels were used depending on our confidence in the product’squality and the extent of feedback we were looking for. The sequence looked somethinglike this:

Canary Channel is used forcode we suspect isn’t fit for release. Like a canary in a coalmine, if itfailed to survive then we had work to do. Canary channel builds are only forthe ultra tolerant user running experiments and not depending on theapplication to get real work done.

Dev Channel is whatdevelopers use on their day-to-day work. All engineers on a product areexpected to pick this build and use it for real work.

Test Channel is the buildused for internal dog food and represents a candidate beta channel build givengood sustained performance.

The Beta Channel or ReleaseChannel builds are the first ones that get external exposure. A build only getsto the release channel after spending enough time in the prior channels that isgets a chance to prove itself against a barrage of both tests and real usage.

This crawl, walk, runapproach gives us the chance to run tests and experiment on our applicationsearly and obtain feedback from real human beings in addition to all theautomation we run in each of these channels every day.

There are analyticalbenefits to this process as well. If a bug is found in the field a tester cancreate a test that reproduces it and run it against builds in each channel todetermine if a fix has already been implemented.

One of the key ways Google achievesgood results with fewer testers than many companies is that we rarely attemptto ship a large set of features at once. In fact, the exact opposite is oftenthe goal: build the core of a product and release it the moment it is useful toas large a crowd as feasible, then get their feedback and iterate. This is whatwe did with Gmail, a product that kept its beta tag for four years. That tagwas our warning to users that it was still being perfected. We removed the betatag only when we reached our goal of 99.99% uptime for a real user’s emaildata. Obviously, quality is a work in progress!

It’s not as cowboy a processas I make it out to be. In fact, in order to make it to what we call the betachannel release, a product must go through a number of other channels and proveits worth. For Chrome, a product I spent my first two years at Google workingon, multiple channels were used depending on our confidence in the product’squality and the extent of feedback we were looking for. The sequence looked somethinglike this:

Canary Channel is used forcode we suspect isn’t fit for release. Like a canary in a coalmine, if itfailed to survive then we had work to do. Canary channel builds are only forthe ultra tolerant user running experiments and not depending on theapplication to get real work done.

Dev Channel is whatdevelopers use on their day-to-day work. All engineers on a product areexpected to pick this build and use it for real work.

Test Channel is the buildused for internal dog food and represents a candidate beta channel build givengood sustained performance.

The Beta Channel or ReleaseChannel builds are the first ones that get external exposure. A build only getsto the release channel after spending enough time in the prior channels that isgets a chance to prove itself against a barrage of both tests and real usage.

This crawl, walk, runapproach gives us the chance to run tests and experiment on our applicationsearly and obtain feedback from real human beings in addition to all theautomation we run in each of these channels every day.

There are analyticalbenefits to this process as well. If a bug is found in the field a tester cancreate a test that reproduces it and run it against builds in each channel todetermine if a fix has already been implemented.How Google TestsSoftware - Part Four