How Google TestsSoftware - Part One

This is the firstin a series of posts on this topic.

The one question I get morethan any other is "How does Google test?" It's been explained in bitsand pieces on this blog but the explanation is due an update. The Googletesting strategy has never changed but the tactical ways we execute it hasevolved as the company has evolved. We're now a search, apps, ads, mobile,operating system, and so on and so forth company. Each of these Focus Areas (aswe call them) have to do things that make sense for their problem domain. As weadd new FAs and grow the existing ones, our testing has to expand and improve.What I am documenting in this series of posts is a combination of what we aredoing today and the direction we are trending toward in the foreseeable future.

Let's begin withorganizational structure and it's one that might surprise you. There isn't anactual testing organization at Google. Test exists within a Focus Area calledEngineering Productivity. Eng Prod owns any number of horizontal and verticalengineering disciplines, Test is the biggest. In a nutshell, Eng Prod is madeof:

1. A product team thatproduces internal and open source productivity tools that are consumed by allwalks of engineers across the company. We build and maintain code analyzers,IDEs, test case management systems, automated testing tools, build systems,source control systems, code review schedulers, bug databases... The idea is tomake the tools that make engineers more productive. Tools are a very large partof the strategic goal of prevention over detection.

2. A services team thatprovides expertise to Google product teams on a wide array of topics includingtools, documentation, testing, release management, training and so forth. Ourexpertise covers reliability, security, internationalization, etc., as well asproduct-specific functional issues that Google product teams might face. Everyother FA has access to Eng Prod expertise.

3. Embedded engineers thatare effectively loaned out to Google product teams on an as-needed basis. Someof these engineers might sit with the same product teams for years, otherscycle through teams wherever they are needed most. Google encourages all itsengineers to change product teams often to stay fresh, engaged and objective.Testers are no different but the cadence of changing teams is left to theindividual. I have testers on Chrome that have been there for several years andothers who join for 18 months and cycle off. Keeping a healthy balance betweenproduct knowledge and fresh eyes is something a test manager has to pay closeattention to.

So this means that testersreport to Eng Prod managers but identify themselves with a product team, likeSearch, Gmail or Chrome. Organizationally they are part of both teams. They sitwith the product teams, participate in their planning, go to lunch with them,share in ship bonuses and get treated like full members of the team. Thebenefit of the separate reporting structure is that it provides a forum fortesters to share information. Good testing ideas migrate easily within Eng Prodgiving all testers, no matter their product ties, access to the best technologywithin the company.

This separation of projectand reporting structures has its challenges. By far the biggest is that testersare an external resource. Product teams can't place too big a bet on them andmust keep their quality house in order. Yes, that's right: at Google it's theproduct teams that own quality, not testers. Every developer is expected to dotheir own testing. The job of the tester is to make sure they have the automationinfrastructure and enabling processes that support this self reliance. Testersenable developers to test.

What I like about thisstrategy is that it puts developers and testers on equal footing. It makes ustrue partners in quality and puts the biggest quality burden where it belongs:on the developers who are responsible for getting the product right. Anotherside effect is that it allows us a many-to-one dev-to-test ratio. Developersoutnumber testers. The better they are at testing the more they outnumber us.Product teams should be proud of a high ratio!

Ok, now we're all friendshere right? You see the hole in this strategy I am sure. It's big enough todrive a bug through. Developers can't test! Well, who am I to deny that? Noamount of corporate kool-aid could get me to deny it, especially coming off myGTAC talk last year where I pretty much made a game of developer vs. tester(spoiler alert: the tester wins).

Google's answer is to splitthe role. We solve this problem by having two types of testing roles at Googleto solve two very different testing problems. In my next post, I'll talk aboutthese roles and how we split the testing problem into two parts.