Test patterns, frustratingly, have no common definitions. Ask 2 people about an “integration test” and you will get three answers. The following are definitions some of us use so that we can speak to each other with a common language.
There are common patterns to show how much of each kind of test is generally recommended. The most used are the Test Pyramid and the Test Trophy. Both are trying to communicate the same thing: design a test suite that is fast, gives you confidence, and is not more expensive to maintain than the value it brings.
Our Testing Principles
- Balance cost and confidence
- Move failure detection as close to the developer as possible
- Increase the speed of running tests
- Aim for CI to take less than 10 minutes. Full test suite should take less than an hour.
Recommended Test Pattern
Most of the tests are integration tests and emphasizes maximizing deterministic test coverage in process with the development cycle, so developers can find errors sooner. E2E & functional tests should primarily focus on happy/critical path and tests that absolutely require a browser/app.
When executing continuous delivery, test code is a first class citizen that requires as much design and maintenance as production code. Flakey tests undermine confidence and should be terminated with extreme prejudice.
|PR Verify, Trunk Verify||Yes||Yes||Yes||Yes||Yes||No||No|
|Test Doubles||Yes||Yes||Yes||Yes||Yes||See Definition||No|
|Network Access||No||No||localhost only||localhost only||No||Yes||Yes|
|File System Access||No||No||No||No||No||No||Yes|
|Database||No||No||localhost only||localhost only||No||Yes||Yes|
General testing best practices are documented here. Best practices specific to test types are documented within each test type page.