Episode 388: Testing vs Monitoring

A user-facing bug causes search results to be unavailable for your service. Someone suggests adding a prober to monitor the service and, if search results are unavailable, notify the team via bug report. Google has plenty of prober options available; if we pick one and use it we’re done, right?

Not necessarily. Just because monitoring could detect a bug does not mean it is the best, or only, solution. For any given bug, you should consider which mixture of monitoring and testing is appropriate. Monitoring and testing each have pros and cons, and solve slightly different problems.

Monitoring observes—and sometimes interacts with—user-facing production systems. Monitoring is useful for detecting:

• Load issues: Real users accessing real services induces load on real servers. The only way to measure the effect of production traffic is to directly measure the servers themselves. Common measurements include QPS, RPC response times, memory usage, and disk usage.
• Service unavailability: A service might become unavailable because (1) the service itself is down, or (2) the services on which it depends are unavailable. Monitoring end-user experiences (e.g., “Does web search return results?”) is a great way to detect and alert about this situation.
• Unanticipated user behavior: Even the most well-designed test scenarios can fail to anticipate real user behavior. Monitoring can inform your quality strategy by observing and measuring real-world behavior.
• Version incompatibility: Different binary versions may interact incompatibly in ways that are hard to detect without production data. Monitoring can detect unanticipated data inconsistencies.
• Data changes: User-facing data can change over time, sometimes in bad ways. Monitoring can statistically characterize data, diff them against previous data state, and alert on outside-of-threshold changes.

Testing isolates components in a non-production environment and verifies components’ behavior. Since it occurs prior to release, it reduces the cost of fixing a bug. Testing is useful for ensuring:

• Functional correctness: A hermetic unit test remains the best way to prove that a small piece of code logic fulfills its interface contract.
• Inter-component compatibility: An integration test is an excellent way to ensure that two components (e.g., a client and a Stubby service) work together properly.

When considering which techniques to employ, review the list above to determine which ones are appropriate. Worried about an individual vendor’s ad inventory suddenly dropping? Monitor ad volume for each vendor! Unsure if the price2value() function handles currency conversions? Write a unit test! Not sure how often users actually log into your system? Monitor login events! A judicious mix of monitoring and testing will speed up development and ensure that fewer bugs reach end users.