E2E Testing

  3 minute read  

End to end tests are typically non-deterministic tests that validate the software system along with its integration with external interfaces. The purpose of end-to-end Test is to exercise a complete production-like scenario. Along with the software system, it also validates batch/data processing from other upstream/downstream systems. Hence, the name “End-to-End”. End to End Testing is usually executed after functional testing. It uses actual production like data and test environment to simulate real-time settings

– Testing Glossary

End to end tests have the advantage of exercising the system in ways that functional tests cannot. However, they also have the disadvantage of being slower to provide feedback, require more state management, constant maintenance, and can fail for reasons unrelated to code defects. As such, it is recommended that they be the smallest number of tests executed.

“E2E Test”

End-to-end tests are segmented into two categories: vertical and horizontal tests.

Vertical E2E Tests

Vertical tests are end to end tests which target features under the control of a single team. Examples of these may be “when I click the heart icon on an item, it’s favorited and that persists across a refresh” or “a user can create a new saved list and add items to it”.

Horizontal E2E Tests

A horizontal test, by contrast, spans multiple teams. An example of this may be going from the homepage through checkout. That involves coordination across the homepage, item page, cart, and checkout teams.

Because of the inherent complexity of horizontal tests (multi-team), they are unsuitable for blocking release pipelines.

  • E2E tests should be the least used due to their cost in run time and in maintenance required.
  • Focus on happy-path validation of business flows
  • E2E tests can fail for reasons unrelated to the coding issues. Capture the frequency and cause of failures so that efforts can be made to make them more stable.
  • Vertical E2E tests should be maintained by the team at the start of the flow and versioned with the component (UI or service).
  • CD pipelines should be optimized for the rapid recovery of production issues. Therefore, horizontal E2E tests should not be used to block delivery due to their size and relative failure surface area.
  • A team may choose to run vertical E2E in their pipeline to block delivery, but efforts must be made to decrease false positives to make this valuable.

Alternate Terms

Integration test and end to end are often used internchangeably.

Resources

Examples

embed:examples/e2e-test.java
Last modified October 13, 2022: Corrected broken links (#8) (d324154)