How Google TestsSoftware - Part Five

Instead of distinguishingbetween code, integration and system testing, Google uses the language ofsmall, medium and large tests emphasizing scope over form. Small tests coversmall amounts of code and so on. Each of the three engineering roles mayexecute any of these types of tests and they may be performed as automated ormanual tests.

Small Tests are mostly (butnot always) automated and exercise the code within a single function or module.They are most likely written by a SWE or an SET and may require mocks and fakedenvironments to run but TEs often pick these tests up when they are trying todiagnose a particular failure. For small tests the focus is on typicalfunctional issues such as data corruption, error conditions and off by oneerrors. The question a small test attempts to answer is does this code do whatit is supposed to do?

Medium Tests can beautomated or manual and involve two or more features and specifically cover theinteraction between those features. I've heard any number of SETs describe thisas "testing a function and its nearest neighbors." SETs drive thedevelopment of these tests early in the product cycle as individual featuresare completed and SWEs are heavily involved in writing, debugging andmaintaining the actual tests. If a test fails or breaks, the developer takescare of it autonomously. Later in the development cycle TEs may perform mediumtests either manually (in the event the test is difficult or prohibitivelyexpensive to automate) or with automation. The question a medium test attemptsto answer is does a set of near neighbor functions interoperate with each otherthe way they are supposed to?

Large Tests cover three ormore (usually more) features and represent real user scenarios to the extentpossible. There is some concern with overall integration of the features butlarge tests tend to be more results driven, i.e., did the software do what theuser expects? All three roles are involved in writing large tests andeverything from automation to exploratory testing can be the vehicle toaccomplish accomplish it. The question a large test attempts to answer is doesthe product operate the way a user would expect?

The actual language ofsmall, medium and large isn’t important. Call them whatever you want. Theimportant thing is that Google testers share a common language to talk aboutwhat is getting tested and how those tests are scoped. When some enterprisingtesters began talking about a fourth class they dubbed enormous every othertester in the company could imagine a system-wide test covering nearly everyfeature and that ran for a very long time. No additional explanation wasnecessary.

The primary driver of whatgets tested and how much is a very dynamic process and varies wildly fromproduct to product. Google prefers to release often and leans toward getting aproduct out to users so we can get feedback and iterate. The general idea isthat if we have developed some product or a new feature of an existing productwe want to get it out to users as early as possible so they may benefit fromit. This requires that we involve users and external developers early in theprocess so we have a good handle on whether what we are delivering is hittingthe mark.

Finally, the mix betweenautomated and manual testing definitely favors the former for all three sizesof tests. If it can be automated and the problem doesn’t require humancleverness and intuition, then it should be automated. Only those problems, inany of the above categories, which specifically require human judgment, such asthe beauty of a user interface or whether exposing some piece of dataconstitutes a privacy concern, should remain in the realm of manual testing.

Having said that, it isimportant to note that Google performs a great deal of manual testing, bothscripted and exploratory, but even this testing is done under the watchful eyeof automation. Industry leading recording technology converts manual tests toautomated tests to be re-executed build after build to ensure minimalregressions and to keep manual testers always focusing on new issues. We alsoautomate the submission of bug reports and the routing of manual testing tasks.For example, if an automated test breaks, the system determines the last codechange that is the most likely culprit, sends email to its authors and files abug. The ongoing effort to automate to within the “last inch of the human mind”is currently the design spec for the next generation of test engineering toolsGoogle is building.

Those tools will behighlighted in future posts. However, my next target is going to revolve aroundThe Life of an SET. I hope you keep reading.
Instead of distinguishingbetween code, integration and system testing, Google uses the language ofsmall, medium and large tests emphasizing scope over form. Small tests coversmall amounts of code and so on. Each of the three engineering roles mayexecute any of these types of tests and they may be performed as automated ormanual tests.

Small Tests are mostly (butnot always) automated and exercise the code within a single function or module.They are most likely written by a SWE or an SET and may require mocks and fakedenvironments to run but TEs often pick these tests up when they are trying todiagnose a particular failure. For small tests the focus is on typicalfunctional issues such as data corruption, error conditions and off by oneerrors. The question a small test attempts to answer is does this code do whatit is supposed to do?

Medium Tests can beautomated or manual and involve two or more features and specifically cover theinteraction between those features. I've heard any number of SETs describe thisas "testing a function and its nearest neighbors." SETs drive thedevelopment of these tests early in the product cycle as individual featuresare completed and SWEs are heavily involved in writing, debugging andmaintaining the actual tests. If a test fails or breaks, the developer takescare of it autonomously. Later in the development cycle TEs may perform mediumtests either manually (in the event the test is difficult or prohibitivelyexpensive to automate) or with automation. The question a medium test attemptsto answer is does a set of near neighbor functions interoperate with each otherthe way they are supposed to?

Large Tests cover three ormore (usually more) features and represent real user scenarios to the extentpossible. There is some concern with overall integration of the features butlarge tests tend to be more results driven, i.e., did the software do what theuser expects? All three roles are involved in writing large tests andeverything from automation to exploratory testing can be the vehicle toaccomplish accomplish it. The question a large test attempts to answer is doesthe product operate the way a user would expect?

The actual language ofsmall, medium and large isn’t important. Call them whatever you want. Theimportant thing is that Google testers share a common language to talk aboutwhat is getting tested and how those tests are scoped. When some enterprisingtesters began talking about a fourth class they dubbed enormous every othertester in the company could imagine a system-wide test covering nearly everyfeature and that ran for a very long time. No additional explanation wasnecessary.

The primary driver of whatgets tested and how much is a very dynamic process and varies wildly fromproduct to product. Google prefers to release often and leans toward getting aproduct out to users so we can get feedback and iterate. The general idea isthat if we have developed some product or a new feature of an existing productwe want to get it out to users as early as possible so they may benefit fromit. This requires that we involve users and external developers early in theprocess so we have a good handle on whether what we are delivering is hittingthe mark.

Finally, the mix betweenautomated and manual testing definitely favors the former for all three sizesof tests. If it can be automated and the problem doesn’t require humancleverness and intuition, then it should be automated. Only those problems, inany of the above categories, which specifically require human judgment, such asthe beauty of a user interface or whether exposing some piece of dataconstitutes a privacy concern, should remain in the realm of manual testing.

Having said that, it isimportant to note that Google performs a great deal of manual testing, bothscripted and exploratory, but even this testing is done under the watchful eyeof automation. Industry leading recording technology converts manual tests toautomated tests to be re-executed build after build to ensure minimalregressions and to keep manual testers always focusing on new issues. We alsoautomate the submission of bug reports and the routing of manual testing tasks.For example, if an automated test breaks, the system determines the last codechange that is the most likely culprit, sends email to its authors and files abug. The ongoing effort to automate to within the “last inch of the human mind”is currently the design spec for the next generation of test engineering toolsGoogle is building.

Those tools will behighlighted in future posts. However, my next target is going to revolve aroundThe Life of an SET. I hope you keep reading.