How Google Tests Software - The Life of a TE

By James Whittaker

The Test Engineer is a newerrole within Google than either SWEs or SETs. As such, it is a role still in theprocess of being defined. The current generation of Google TEs are blazing atrail which will guide the next generation of new hires for this role. It isthe process that is emerging as the best within Google that we present here.

Not all products require theservices of a TE. Experimental efforts and early stage products without a well-definedmission or user story are certainly projects that won’t get a lot of TEattention. If the product stands a good chance of being cancelled (in the sensethat as a proof of concept it fails to pass muster) or has yet to engage usersor have a well defined set of features, testing it is largely something thatshould be done by the people developing it.

Even if it is clear that aproduct is going to get shipped, Test Engineers have little to do early in thedevelopment cycle when features are still in flux and the final feature listand scope is undetermined. Overinvesting in testing too early can mean a lot ofthings get thrown away. Likewise, early testing planning requires fewer testengineers than later cycle exploratory testing when the product is close tofinal form and the hunt for missed bugs has a greater urgency.

The trick in staffing aproject with Test Engineers has to do with risk and return on investment. Riskto the customer and to the enterprise means more testing effort and requiresmore TEs. But that effort needs to be in proportion to the potential return. Weneed the right number of TEs and we need them to engage at the right time andwith the right impact.

Once engaged, TEs do nothave to start from scratch. There is a great deal of test engineering andquality-oriented work performed by SWEs and SETs which is the starting pointfor additional TE work. The initial engagement of the TE is to decide thingssuch as:

· Where are the weak points in the software?

· What are the security, privacy, performance andreliability concerns?

· Do all the primary user scenarios work as expected?For all international audiences?

· Does the product interoperate with other products(hardware and software)?

· In the event of a problem, how good are thediagnostics?

All of this combines tospeak to the risk profile of releasing the software in question. TEs don’tnecessarily do all of this work, but they ensure that it gets done and theyleverage the work of others is assessing where additional work is required.Ultimately, test engineers are paid to protect users and the business from baddesign, confusing UX, functional bugs, security and privacy issues and soforth. At Google, TEs are the only people on a team whose full-time job is tolook at the product or service holistically for weak points. As such, the lifeof a Test Engineer is much less prescriptive and formalized than that of anSET. TE’s are asked to help on projects in all stages of readiness: everythingfrom the idea stage to version 8, or even watching over a deprecated or“mothballed” project. Often, a single TE will even span multiple projectsparticularly those with specialty type skills like security.

Obviously, the work of a TEvaries greatly depending on the project. Some TE’s spend much of their timeprogramming, much like an SET, but with more of a focus on end-to-end userscenarios. Other TE's take existing code and designs determine failure modesand look for errors that will cause those failures. In such a role a TE might modifycode but not create it from scratch. TE's must be more systematic and thoroughin their test planning and completeness with a focus on the actual usage andsystem experience. TE's excel at dealing with ambiguity in requirements and atreasoning and communicating about fuzzy problems.

Successful TEs accomplishall this while navigating the sensitivities and sometimes strong personalitiesof the development and product team members. When weak points are found, testengineers happily break the software, and drive to get these issues resolvedwith the SWEs, PMs, and SETs.

Such a job description is afrightening prospect given the mix of technical skill, leadership, and deepproduct understanding and without proper guidance it is a role in which manywould expect to fail. But at Google a strong community of test engineers hasemerged to counter this. Of all job functions, the TE role is perhaps the bestpeer supported role in the company and the insight and leadership required toperform it successfully means that many of the top test managers in the companycome from the TE ranks.

There is a fluidity to thework of a Google Test Engineer that belies any prescriptive process forengagement. TE’s can enter a project at any point and must assess the state ofthe project, code, design, and users quickly and decide what to focus on first.If the project is just getting started, test planning is often the first orderof business. Sometimes TEs are pulled in late in the cycle to evaluate whethera project is ready for ship or if there are any major issues before an early‘beta’ goes out. If they are brought into a newly acquired application or onein which they have little prior experience, they will often start doing someexploratory testing with little to no planning. Sometimes projects haven’t beenreleased for quite a while and just need some touchups/security fixes, or UXupdates—calling for an even different approach. One size rarely fits all forTEs at Google.