Designing Tests

  2 minute read  

Test patterns, frustratingly, have no common definitions. Ask 2 people about an “integration test” and you will get three answers. The following are definitions some of us use so that we can speak to each other with a common language.

There are common patterns to show how much of each kind of test is generally recommended. The most used are the Test Pyramid and the Test Trophy. Both are trying to communicate the same thing: design a test suite that is fast, gives you confidence, and is not more expensive to maintain than the value it brings.

Our Testing Principles

  • Balance cost and confidence
  • Move failure detection as close to the developer as possible
  • Increase the speed of running tests
    • Aim for CI to take less than 10 minutes. Full test suite should take less than an hour.

Most of the tests are integration tests and emphasizes maximizing deterministic test coverage in process with the development cycle, so developers can find errors sooner. E2E & functional tests should primarily focus on happy/critical path and tests that absolutely require a browser/app.

When executing continuous delivery, test code is a first class citizen that requires as much design and maintenance as production code. Flakey tests undermine confidence and should be terminated with extreme prejudice.

Testing Matrix

Feature Static Unit Integration Functional Visual Regression Contract E2E
Deterministic Yes Yes Yes Yes Yes No No
PR Verify, Trunk Verify Yes Yes Yes Yes Yes No No
Break Build Yes Yes Yes Yes Yes No No
Test Doubles Yes Yes Yes Yes Yes See Definition No
Network Access No No localhost only localhost only No Yes Yes
File System Access No No No No No No Yes
Database No No localhost only localhost only No Yes Yes

Testing Anti-pattern

“Ice cream cone testing” is the anti-pattern where the most expensive, fragile, non-deterministic tests are prioritized over faster and less expensive deterministic tests because it “feels” right.

Google Test Blog: Just Say No to More End-to-End Tests

Testing Best Practices

General testing best practices are documented here. Best practices specific to test types are documented within each test type page.

Test Pattern Resources

Testing Best Practices

Contract Testing

E2E Testing

Functional Testing

Testing Terms Glossary

Integration Testing

Static Testing

Test Doubles

Unit Testing

Last modified October 13, 2022: Corrected broken links (#8) (d324154)