# Dojo Consortium > Production tested playbooks and methods for delivering software better ## Definition of Done URL: https://dojoconsortium.org/docs/workflow-management/definition-of-done/ Description: Team agreement on conditions that must be met before work is considered complete, ensuring consistent quality standards Is it DONE, DONE DONE, or is it DONE DONE DONE? All teams need a Definition of Done. The Definition of Done is an agreement made between the team that a unit of work isn’t complete without meeting certain conditions. Recommended Practices We use the Definition of Done most commonly for user stories. The team and product owner must agree that the story has met all criteria for it to be considered done. A definition of done can include anything a team cares about, but must include these criteria: All tests passed All acceptance criteria have been met Code reviewed by team member and merged to trunk Demoed to team/stakeholders as close to prod as possible All code associated with the acceptance criteria deployed to production Once your team has identified all criteria that a unit of work needs to be considered done, you must hold yourself accountable to your Definition of Done. Value As a development team, we want to understand our team’s definition of done, so that we can ensure a unit of work is meeting the criteria acceptable for it to be delivered to our customers. Acceptance Criteria Identify what your team cares about as a Definition of Done. Use your Definition of Done as a tool to ensure quality stories are being released into production. Revisit and evaluate your Definition of Done. --- ## Definition of Ready URL: https://dojoconsortium.org/docs/work-decomposition/definition-of-ready/ Description: Team-agreed criteria that defines when work is ready to begin, helping manage uncertainty and set clear expectations Is it REALLY Ready? A Definition of Ready is a set of criteria decided by the team that defines when work is ready to begin. The goal of the Definition of Ready to help the team decide on the level of uncertainty that they are comfortable with taking on with respect to their work. Without that guidance, any work is fair game. That is a recipe for confusion and disaster. Recommended Practices When deciding on a Definition of Ready, there are certain minimum criteria that should always be there. These are: Description of the value the work provides (Why do we want to do this?) Testable Acceptance Criteria (When do we know we’ve done what we need to?) The team has reviewed and agreed the work is ready (Has the team seen it?) However, the context of a team can make many other criteria applicable. Other criteria could include: Wireframes for new UI components Contracts for APIs/services we depend on All relevant test types identified for subtasks Team estimate of the size of the story is no more than 2 days The Definition of Ready is a living document that should evolve over time as the team works to make their delivery system more predictable. The most important thing is to actually enforce the Definition of Ready. If it’s not enforced, it’s completely useless. If any work in “Ready to Start” does not meet the Definition of Ready, move it back to the Backlog until it is refined. Any work that is planned for a sprint/iteration must meet the Definition of Ready. Do not accept work that isn’t ready! If work needs to be expedited, it needs to go through the same process. (Unless there is immediate production impact, of course) Tips Using Behavior Driven Development is one of the best ways to define testable acceptance criteria. Definition of Ready is also useful for support tickets or other types of work that the team can be responsible for. It’s not just for development work! It’s up to everyone on the team, including the … --- ## Getting Started with Continuous Delivery URL: https://dojoconsortium.org/docs/cd/getting-started/ Description: Practical first steps to begin your Continuous Delivery journey This guide provides actionable steps teams can take in their first week to begin implementing Continuous Delivery practices. Start small, measure progress, and build momentum. Before You Begin Continuous Delivery is a journey, not a destination. You don’t need to have everything perfect before you start. Focus on making incremental improvements and learning from each change. Prerequisites A source code repository with your application A basic build process (even if manual) At least one deployed environment Team buy-in to try new practices Week 1: Foundation Day 1: Establish Your Baseline Action: Measure your current state Before improving, understand where you are. Capture these baseline metrics: Current State Assessment Development: - How long from starting work to merging code? _____ days - How many branches exist right now? _____ - How long do branches live before merging? _____ days - How often does the trunk break? _____ times/week Delivery: - How long from code merge to production? _____ days/weeks - How many manual steps in deployment? _____ - What % of deployments require hotfixes? _____% - How long to restore service after failure? _____ hours Why this matters: You can’t improve what you don’t measure. These numbers will guide your improvement efforts and prove progress. Day 2: Define Your Working Agreement Action: Create a CI Working Agreement Have a team discussion and agree to start with these practices: CI Working Agreement (v1) We agree to: ✓ Merge code to trunk at least once per day ✓ Keep branches alive less than 24 hours ✓ Fix broken builds before starting new work ✓ Include automated tests with every change ✓ Review pull requests within 2 hours ✓ Prioritize completing in-progress work over starting new work Starting: [DATE] Review: [DATE + 2 weeks] Tip Start with what feels achievable, then tighten the agreement as you improve. It’s better to keep a loose agreement than break a strict one. Day 3: Automate Your Build Action: … --- ## Glossary URL: https://dojoconsortium.org/docs/reference/glossary/ Description: Key terms and definitions used throughout the Continuous Delivery documentation Continuous Delivery Continuous Deployment Continuous Integration Hard Dependency Soft Dependency Story Points Toil Unplanned Work Vertical Sliced Story WIP Continuous Delivery The ability to deliver the latest changes to production on demand. Continuous Deployment Delivering the latest changes to production as they occur. Continuous Integration Continuous integration requires that every time somebody commits any change, the entire application is built and a comprehensive set of automated tests is run against it. Crucially, if the build or test process fails, the development team stops whatever they are doing and fixes the problem immediately. The goal of continuous integration is that the software is in a working state all the time. Continuous integration is a practice, not a tool. It requires a degree of commitment and discipline from your development team. You need everyone to check in small incremental changes frequently to mainline and agree that the highest priority task on the project is to fix any change that breaks the application. If people don’t adopt the discipline necessary for it to work, your attempts at continuous integration will not lead to the improvement in quality that you hope for. – “Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation.” - Jez Humble & David Farley You can find recommended practices for CI at MimimumCD.org Hard Dependency A hard dependency is something that must be in place before a feature is delivered. In most cases, a hard dependency can be converted to a soft dependency with feature flags. Soft Dependency A soft dependency is something that must be in place before a feature can be fully functional, but does not block the delivery of code. Story Points A measure of the relative complexity of delivering a story. Historically, 1 story point was 1 “ideal day”. An ideal day is a day where there are no distractions, the code is flowing, and we … --- ## Testing Terms Glossary URL: https://dojoconsortium.org/docs/testing/glossary/ Description: Standardized definitions for testing terms to establish ubiquitous language and reduce ambiguity in testing discussions Testing terms and they are notoriously overloaded. If you ask 3 people what integration testing means you will get 4 different answers. This ambiguity within an organization slows down the engineering process as the lack of ubiquitous language causes communication errors. For us to help each other improve our quality processes, it is important that we align on a common language. In doing so, we understand that many may not agree 100% on the definitions we align to. That is ok. It is more important to be aligned to consensus than to be 100% in agreement. We’ll iterate and adjust as needed. Note: Our definitions are based on the following sources: Testing Categories by Martin Fowler The Practical Test Pyramid by Ham Vocke xUnit Test Patterns * Refactoring Test Code by Gerard Meszaros Glossary Deterministic Test A deterministic test is any test that always returns the same results for the same beginning state and action. Deterministic tests should always be able to run in any sequence or in parallel. Only deterministic tests should be executed in a CI build or automatically block delivery during CD. Non-deterministic Test A non-deterministic test is any test that may fail for reasons unrelated to adherence to specification. Reasons for this could include network instability, availability of external dependencies, state management issues, etc. Static Test A static test is a test that evaluates non-running code against rules for known good practices to check for security, structure, or practice issues. Unit Test Unit tests are deterministic tests that exercise a discrete unit of the application, such as a function, method, or UI component, in isolation to determine whether it behaves as expected. More on Unit Testing Integration Test An integration test is a deterministic test to verify how the unit under test interacts with other units without directly accessing external sub-systems. For the purposes of clarity, “integration test” is not a test that … --- ## Metrics Quickstart URL: https://dojoconsortium.org/docs/metrics/metrics-quickstart/ Description: Set up essential CD metrics in one day and start improving delivery performance This guide helps you quickly implement the minimum set of metrics needed to measure and improve your Continuous Delivery performance. Start tracking today, improve tomorrow. Why Metrics Matter Goodhart's Law Without metrics, improvement is guesswork. Metrics help you: ✅ Identify bottlenecks in your delivery process ✅ Measure improvement over time ✅ Make data-driven decisions about where to focus ✅ Demonstrate value to leadership ✅ Prevent regression when optimizing Critical Use metrics in groups, never alone. Optimizing a single metric leads to unintended consequences. Always use offsetting metrics to maintain balance. The Essential Four Metrics Start with these four DORA metrics that predict delivery performance: Metric Good Target Purpose Development Cycle Time < 2 days Measure delivery speed Deployment Frequency Multiple/day Measure delivery throughput Change Failure Rate < 5% Measure quality Mean Time to Repair < 1 hour Measure recovery speed These four metrics balance speed (cycle time, deployment frequency) with stability (change failure rate, MTTR). Day 1: Start Tracking Step 1: Deployment Frequency (15 minutes) What it measures: How often you deploy to production Simplest implementation: # Add to your deployment script #!/bin/bash # deploy.sh DEPLOY_TIME=$(date -u +"%Y-%m-%dT%H:%M:%SZ") SERVICE_NAME="my-service" # Your deployment logic here kubectl apply -f deployment.yaml # Log the deployment echo "${DEPLOY_TIME},${SERVICE_NAME},${VERSION}" >> /var/log/deployments.csv # Or send to metrics service curl -X POST https://metrics.example.com/deployments \ -d "{\"service\":\"${SERVICE_NAME}\",\"timestamp\":\"${DEPLOY_TIME}\",\"version\":\"${VERSION}\"}" Query deployment frequency: # Deployments per day (last 30 days) cat /var/log/deployments.csv | \ awk -F',' '{print $1}' | \ cut -d'T' -f1 | \ sort | uniq -c | \ awk '{total+=$1; count++} … --- ## From Roadmap to User Story URL: https://dojoconsortium.org/docs/work-decomposition/program-to-user/ Description: A guide to aligning priorities and breaking down work across multi-team products Aligning priorities across multi-team products can be challenging. This guide outlines how to effectively break down work from program-level roadmaps to team-level user stories. Program Roadmap Key Point Establishing and understanding goals and priorities is crucial for an effective work breakdown process. Program Roadmap Stakeholders and leadership teams must define high-level initiatives and their priorities Work can then be dispersed among product teams Leadership teams can be composed of a core group of product owners Product Roadmap The program roadmap should break down into the product roadmap, which includes the prioritized list of epics for each product. Product Vision The leadership team should define: Product vision Roadmap Dependencies for each product Team Backlog The team backlog should comprise the prioritized epics from the product roadmap. Feature Backlog Effective Work Breakdown The core group needed to effectively break down high-level requirements includes: Product owners Tech leads Project managers Product teams should use processes effective for Work Decomposition to break down epics into: Smaller epics Stories Tasks --- ## Static Testing URL: https://dojoconsortium.org/docs/testing/static/ Description: Code analysis tools that evaluate non-running code for security vulnerabilities, complexity, and best practice violations A static test is a test that evaluates non-running code against rules for known good practices to check for security, structure, or practice issues. – Testing Glossary Static code analysis has many key purposes. It warns of excessive complexity in the code that will degrade the ability to change it safely. Identifies issues that could expose vulnerabilities Shows anti-patterns that violate good practices Alerts to issues with dependencies that may prevent delivery, create a vulnerability, or even expose the company to lawsuits. It catches errors Principles When implementing any test, the test should be designed to provide alerts as close to the moment of creation as possible. Static analysis, many scans can be run realtime in IDEs. Others during the build or as a pre-commit scan. Others require tooling that can only be used on the CI server. Whatever the test, drive it left. Recheck everything on CI while verifying HEAD Types of static tests Linting: This automates catching of common errors in code and the enforcement of best practices Formatting: Enforcement of code style rules. It removes subjectivity from code reviews Complexity: Are code blocks too deep or too long? Complexity causes defects and simple code is better. Type checking: Type checking can be a key validation to prevent hard to identify defects replacing certain classes of tests and logic otherwise required (e.g. unit tests validating internal APIs) Security: Checking for known vulnerabilities and coding patterns that provide attack vectors are critical Dependency scanning : Are your dependencies up to date? Has the dependency been hijacked? Are there known security issues in this version that require immediate resolution? Is it licensed appropriately? Recommended Best Practices IDE plugins to identify problems in realtime Pre-commit hooks to prevent committing problems Verification during PR and during the CI build on the HEAD to verify that earlier verification happened and was effective. … --- ## Testing Quickstart URL: https://dojoconsortium.org/docs/testing/testing-quickstart/ Description: Get your test suite to production-ready in one week A practical guide to building a fast, reliable test suite that gives confidence without slowing down delivery. Focus on deterministic tests that run in CI and provide rapid feedback. The Goal Build a test suite that: ✅ Runs in under 10 minutes ✅ Is deterministic (same result every time) ✅ Catches real bugs before production ✅ Provides fast feedback to developers ✅ Doesn’t require heroic maintenance Before You Begin The Anti-Pattern to Avoid Ice Cream Cone Anti-Pattern Ice Cream Cone Testing = Lots of slow, fragile E2E tests, minimal fast unit/integration tests Why this fails: E2E tests are slow (minutes per test) E2E tests are non-deterministic (flaky) Debugging E2E failures is time-consuming Developers stop trusting the tests The Right Pattern Most tests should be integration tests - fast, deterministic, testing real interactions without external dependencies. See Test Patterns for the full testing matrix. Week 1 Action Plan Day 1: Audit Your Current Tests Action: Categorize and time your existing tests # Run your test suite and capture metrics npm test -- --verbose --timing # or mvn test -DreportFormat=plain Create a test inventory: Test Inventory Unit Tests: Count: _____ Time: _____ seconds Failures (last week): _____ Integration Tests: Count: _____ Time: _____ seconds Failures (last week): _____ E2E Tests: Count: _____ Time: _____ minutes Failures (last week): _____ Total CI Time: _____ minutes Flaky test rate: _____% Red flags: 🚩 Total CI time > 10 minutes 🚩 Flaky test rate > 1% 🚩 More E2E tests than integration tests 🚩 Tests that require deployed environments Day 2: Fix or Delete Flaky Tests Action: Zero tolerance for non-deterministic tests Flaky tests destroy confidence. They must be fixed immediately or deleted. Identify flaky tests: # Run tests 10 times, capture failures for i in {1..10}; do npm test 2>&1 | tee test-run-$i.log done # Analyze failures grep "FAIL" test-run-*.log | sort | uniq -c Common causes of flakiness: Cause … --- ## Work Decomposition URL: https://dojoconsortium.org/docs/work-decomposition/work-breakdown/ Description: A guide to effectively breaking down work into manageable, deliverable units Effective work decomposition is crucial for delivering value faster with less rework. This guide outlines the process and best practices for breaking down work from ideas to tasks. Prerequisites Before implementing the work breakdown flow, ensure your team has: Definition of Ready Definition of Done Backlog refinement cadence with appropriate team members and stakeholders Work Breakdown Process Work Breakdown Flow Goal Decompose work into small batches that can be delivered frequently, multiple times a week. Key Tips for Work Decomposition Known poor quality should not flow downstream Plan refinement meetings when people are mentally alert Good acceptance criteria come from good communication Focus on outcomes, not volume, during refinement Workflow Stages of Work Breakdown 1. Intake/Product Ideas Ideas become epics with defined outcomes, clear goals, and value Epics become a list of features Common struggles: Unclear requirements Unclear goals 2. Refining Epics/Features into Stories Stories are observable changes with clear acceptance criteria, completable in less than two days. Typical problems: Stories are too big or complex Stories lack testable acceptance criteria Lack of dependency knowledge Managing research tasks 3. Refining Stories into Development Tasks Tasks are independently deployable changes, mergeable to trunk daily Breaking stories into tasks allows teams to swarm work and deliver value faster Teams need to understand what makes a good task Measuring Success Key Metric Track the team’s Development Cycle Time to judge improvements in decomposition. Ideal characteristics: Stories take 1-2 days to deliver No rework No delays waiting for explanations No dependencies on other stories or teams --- ## Behavior Driven Development URL: https://dojoconsortium.org/docs/work-decomposition/behavior-driven-development/ Description: Collaborative process for defining feature behaviors through declarative, testable acceptance criteria that serve as Definition of Done Behavior Driven Development is the collaborative process where we discuss the intent and behaviors of a feature and document the understanding in a declarative, testable way. These testable acceptance criteria should be the Definition of Done for a user story. BDD is not a technology or automated tool. BDD is the process of defining the behavior. We can then automate tests for those behaviors. Example: Feature: I need to smite a rabbit so that I can find the Holy Grail Scenario: Use the Holy Hand Grenade of Antioch Given I have the Holy Hand Grenade of Antioch When I pull the pin And I count to 3 But I do not count to 5 And I lob it towards my foe And the foe is naughty in my sight Then my foe should snuff it Recommended Practices Gherkin is the domain specific language that allows acceptance criteria to be expressed in “Arrange, Act, Assert” in a way that is understandable to all stakeholders. Example: Feature: As an hourly associate I want to be able to log my arrival time so that I can be paid correctly. Scenario: Clocking in Given I am not clocked in When I enter my associate number Then my arrival time will be logged And I will be notified of the time Scenario: Clocking out Given I am clocked in When I enter my associate number And I have been clocked in for more than 5 minutes Then I will be clocked out And I will be notified of the time Scenario: Clocking out too little time Given I am clocked in When I enter my associate number And I have been clocked in for less than 5 minutes Then I will receive an error Using Acceptance Criteria to Negotiate and Split With the above criteria, it may be acceptable to remove the time validation and accelerate the delivery of the time logging ability. After delivery, we may learn that the range validation isn’t required. If true, we’ve saved money and time by NOT delivering unneeded features. First, we deliver the ability to clock in and see if we really do need the ability to verify. Feature: As an … --- ## Metrics Cheat Sheet URL: https://dojoconsortium.org/docs/metrics/metrics-cheat-sheet/ Description: Quick reference guide for key CD metrics with targets and improvement strategies Organizational Metrics These metrics are important for teams and management to track the health of the delivery system Metric Meaning Goal of Measuring Guardrail Metrics Integration/Merge Frequency How frequently code changes are integrated to the trunk for testing Reduce the size of change to improve quality and reduce risk Defect Rates should not increase Build Cycle Time Total duration from commit to production delivery Improve the ability to deliver changes to improve feedback and reduce MTTR Defect Rates should not increase Change Fail % The % of production deploys that are reverted Improve the upstream quality processes Development Cycle Time should not increase Code Inventory Lines of code added or removed that have not been delivered to production Reduce the amount of code inventory and move closer to Just In Time delivery. Change Fail % & Defect Rate should not increase Defect Rate Number of defects created during a set interval Improve the quality processes in the delivery flow Delivery Frequency should not reduce Development Cycle Time Time from when a story is started until marked “done” Reduce the size of work to improve the feedback from the end user on the value of the work and to improve the quality of the acceptance criteria and testing Defect Rate should not increase MTTR The time from when customer impact begins until it is resolved Improve the stability and resilience of both the application and the system of delivery Quality should not decrease Delivery Frequency The frequency that changes are delivered to production Reduce the size of delivered change, improve the feedback loop on quality and increase the speed of value delivery. Defect Rates should not degrade Work in Progress The number of items in progress on the team relative to the size of the team Reduce the number of items in progress so that the team can focus on completing work vs/ being busy. Delivery frequency should not degrade Team Metrics These metrics should only … --- ## Common Blockers URL: https://dojoconsortium.org/docs/cd/cd-problems/ Description: Common issues teams encounter when implementing Continuous Delivery and how to resolve them The following are very frequent issues that teams encounter when working to improve the flow of delivery. Work Breakdown Stories without testable acceptance criteria All stories should be defined with declarative and testable acceptance criteria. This reduces the amount of waiting and rework once coding begins and enables a much smoother testing workflow. Acceptance criteria should define “done” for the story. No behavior other than that specified by the acceptance criteria should be implemented. This ensures we are consistently delivering what was agreed to. Stories too large It’s common for teams using two week sprints to have stories that require five to ten days to complete. Large stories hide complexity, uncertainty, and dependencies. Stories represent the smallest user observable behavior change. To enable rapid feedback, higher quality acceptance criteria, and more predictable delivery, Stories should require no more than two days for a team to deliver. No definition of “ready” Teams should have a working agreement about the definition of “ready” for a story or task. Until the team agrees it has the information it needs, no commitments should be made and the story should not be added to the “ready” backlog. Definition of Ready - Story - Acceptance criteria aligned with the value statement agreed to and understood. - Dependencies noted and resolution process for each in place - Spikes resolved. - Sub-task - Contract changes documented - Component acceptance tests defined No definition of “Done” Having an explicit definition of done is important to keeping WIP low and finishing work. Definition of Done - Sub-task - Acceptance criteria met - Automated tests verified - Code reviewed - Merged to Trunk - Demoed to team - Deployed to production - Story - PO Demo completed - Acceptance criteria met - All tasks "Done" - Deployed to production Team Workflow Assigning tasks for the sprint Work … --- ## Limiting Work in Progress URL: https://dojoconsortium.org/docs/workflow-management/limiting-wip/ Description: Reduce context switching and improve flow by limiting started-but-unfinished work, helping teams focus on collaboration and completion Why Limit WIP? Work in Progress is defined as work that has started but is not yet finished. Limiting WIP helps teams reduce context switching, find workflow issues, and keep teams focused on collaboration and finishing work. How do we limit WIP? Start with one lane on your board. Set your WIP limit to N+2 (“N” being the number of people contributing to that lane) Continue setting WIP lower. Once the WIP limit is reached, no more cards can enter that lane until one exits. Capacity Utilization There is a direct correlation between WIP and capacity utilization. Attempting to load people and resources to 100% capacity utilization creates wait times. Unpredictable events equal variability, which equals capacity overload. The more individuals and resources used, the higher the cost and risk. In order to lessen work in progress, be aggressive in prioritization, push back when necessary, and set hard WIP limits. Select a WIP limit that is doable but challenges you to say no some of the time. Conflicting Priorities When we start a new task before finishing an older task, our work in progress goes up and things take longer. Business value that could have been realized sooner gets delayed because of too much WIP. Be wary of falling back into the old habit of starting everything because of the pressure to say yes to everything. Look at priority ways of working: Assigned priority Cost of delay First-in, first-out Tips Swarming Stories Having more than one person work on a task at the same time avoids situations where team understanding is mostly limited to a subset of what’s being built. With multiple people involved early, there is less chance that rework will be needed later. By having more than one developer working on a task, you are getting a real-time code review. Story assignment Visually distinguish important information. Who’s working on what? Has this work been in progress for too long? Is this work blocked from progressing? Have we reached our … --- ## Unit Testing URL: https://dojoconsortium.org/docs/testing/unit/ Description: Fast, deterministic tests that verify individual functions, methods, or components in isolation with test doubles for dependencies Unit tests are deterministic tests that exercise a discrete unit of the application, such as a function, method, or UI component, in isolation to determine whether it behaves as expected. – Testing Glossary When testing the specs of functions, prefer testing public API (methods, interfaces, functions) to private API: the spec of private functions and methods are meant to change easily in the future, and unit-testing them would amount to writing a Change Detector Test, which is an anti-pattern. The purpose of unit tests are to: Verify the functionality of a unit (method, class, function, etc.) in isolation Good for testing hi-complexity logic where there may be many permutations (e.g. business logic) Keep Cyclomatic Complexity low through good separations of concerns and architecture Principles Unit tests are low-level and focus on discrete units of the application All dependencies are typically replaced with test-doubles to remove non-determinism Unit tests are fast to execute Test Suite is ran after every code change Recommended Best Practices Run a subset of your test suite based on the part of the code your are currently working on Following TDD practices plus the watch functionality of certain testing frameworks is an easy way to achieve this Pre-commit hooks to run the test suite before committing code to version control Verification during PR and during the CI build on the HEAD to verify that earlier verification happened and was effective. Discourage disabling of static tests (e.g. skipping tests, ignoring warnings, ignoring code on coverage evaluation, etc) Write custom rules (lint, formatting, etc) for common code review feedback Resources Unit Testing by Martin Fowler xUnit Patterns Examples JavaScript Java // Example from lodash describe('castArray', () => { it('should wrap non-array items in an array', () => { const values = falsey.concat(true, 1, 'a', { a: 1 }); const expected = lodashStable.map(values, (value) => … --- ## Value Stream Mapping URL: https://dojoconsortium.org/docs/reference/value-stream-mapping/ Description: A guide to conducting a Value Stream Mapping Workshop to optimize your development process. The Value Stream Mapping Workshop uncovers all steps from idea conception to production, aiming to identify removable steps, bottlenecks, and high-defect areas. Overview Value Stream Mapping helps teams: Identify and remove unnecessary steps Uncover waiting periods between steps Highlight steps with high defect rates The outcome guides the design of an improved value stream, prioritizing changes to reduce waste in the current flow. Prerequisites An established process for value delivery (for a “to be” value stream) Participation from all stakeholders in the value stream Understanding of key terms: Wait time/non-value time Process time/value add time Percent Complete/Accurate (%C/A) Recommended Practices Start mapping from delivery and move backward to ensure no steps are missed. Process 1. Identify the Source Example Team Demo For each source of Requests, determine: Average process time Involved stakeholders Percentage of work rejected by the next step Process Step Example 2. Identify Rework Loops Rework loops are interruptions where steps need correction. Rework Loop Example 3. Identify Wait Time Calculate wait time between steps, considering your team’s cadence. Wait Time Example Outcomes Process time/wait time of your flow Visual representation of the value stream(s) Potential constraints (represented as kaizen bursts) Complete Value Stream Map Tips Regularly review and update the value stream map Consider all potential flows for team processes Value Proposition Understanding how to value stream map team processes helps identify delivery constraints and improvement opportunities. Acceptance Criteria Value stream all processes associated with delivering value Create actionable improvement items from the exercise Further Reading Value Stream Mapping: How to Visualize Work and Align Leadership for Organizational Transformation Flow Engineering: From Value Stream Mapping to Effective Action --- ## Cloud Native Checklist URL: https://dojoconsortium.org/docs/reference/cloud-checklist/ Description: A comprehensive checklist for cloud-native architecture principles and practices Cloud Native checklist Principles and Practices Small, autonomous, highly-cohesive services Hypermedia-driven service interactions Modeled around business concepts Hide internal implementation details Decentralize everything Deploy independently Isolate failure Highly observable Culture of automation References Cloud Native checklist Capability Yes / No Domain Context diagram current with dependencies shown Exception logging Logs stream or self-purge Dynamically configurable log levels Database connections self-heal Dependency connections self-heal Service auto-restarts on failure Automated resource and performance monitoring Have NFRs & SLAs defined for each service Automated alerting for SLAs and NFRs No manual install steps Utilize Correlation ID Load balanced Automated smoke tests after each deployment Heartbeat responds in less than 1 minute after startup No start-up ordering required Minimal critical dependencies Graceful degradation for non-critical dependencies Circuit breakers and request throttles in place Principles and Practices While practices may change over time, principles are expected to be less volatile. Small, autonomous, highly-cohesive services Prefer event-driven, asynchronous communications between services. Prefer eventual consistency / replication of select data elements over shared data structures. Be cautious about creating shared binary dependencies across services. Services are able to be checked out and run locally using embedded DBs, and/or mocked endpoint dependencies as necessary. Hypermedia-driven service interactions Model resources on the domain. Use embedded links to drive resource state transitions. HATEOAS Reference Modeled around business concepts Produce a system context diagram to understand your system boundaries. Consider following c4 architecture diagramming techniques. Follow Domain Driven Design practices to understand your domain early in development, and model your domain in your code. Use bounded contexts to … --- ## Integration Testing URL: https://dojoconsortium.org/docs/testing/integration/ Description: Deterministic tests that verify how units interact together or with external systems using test doubles for non-deterministic dependencies An integration test is a deterministic test to verify how the unit under test interacts with other units without directly accessing external sub-systems. For the purposes of clarity, “integration test” is not a test that broadly integrates multiple sub-systems. That is an E2E test. – Testing Glossary Some examples of an integration test are validating how multiple units work together (sometimes called a “sociable unit test”) or validating the portion of the code that interfaces to an external network sub-system while using a test double to represent that sub-system. Validating the behavior of multiple units with no external sub-systems Validating the portion of the code that interfaces to an external network sub-system When designing network integration tests, it’s recommended to also have contract tests running asynchronously to validate the service test doubles. Recommended Best Practices Integration tests provide the best balance of speed, confidence, and cost when building tests to ensure your system is properly functioning. The goal of testing is to give developers confidence when refactoring, adding features or fixing bugs. Integration tests that are decoupled from the implementation details will give you this confidence without giving you extra work when you refactor things. Too many unit tests, however, will lead to very brittle tests. If you refactor code (i.e. change the implementation w/out changing the functionality) the goal should be to NOT break any tests and ideally not even touch them at all. If lots of tests are breaking when you refactor, it’s probably a sign of too many unit tests and not enough integration tests. Tests should be written from the perspective of how the actor experiences it. Avoid hasty abstractions. Duplication in tests is not the enemy. In fact, it’s often better to have duplicated code in tests than it is to have complex abstractions. Tests should be damp, not DRY. Design tests … --- ## Pipeline & Application Architecture URL: https://dojoconsortium.org/docs/cd/delivery-system-improvement-journey/ Description: A guide to improving your delivery pipeline and application architecture for Continuous Delivery This guide provides steps and best practices for improving your delivery pipeline and application architecture. Please review the CD Getting Started guide for context. 1. Build a Deployment Pipeline The first step is to create a single, automated deployment pipeline to production. Human intervention should be limited to approving stage gates where necessary. Entangled Architecture - Requires Remediation Entangled Architecture Characteristics No clear ownership of components or quality Delayed quality signal Difficult to implement Continuous Delivery Common Entangled Practices Team Structure: Feature teams focused on cross-cutting deliverables Development Process: Long-lived feature branches Branching: Team branches with daily integration to trunk Testing: Inverted test pyramid common Pipeline: Focus on establishing reliable build/deploy automation Deploy Cadence / Risk: Extended delivery cadence, high risk Entangled Improvement Plan Find architectural boundaries to divide sub-systems between teams, creating product teams. This will realign to a tightly coupled architecture. Tightly Coupled Architecture - Transitional Tightly Coupled Architecture Characteristics Changes in one part can affect other parts unexpectedly Sub-assemblies assigned to product teams Requires a more complex integration pipeline Common Tightly Coupled Practices Team Structure: Product teams focused on decoupling sub-systems Development Process: Continuous integration Branching: Trunk-Based Development Testing: Developer Driven Testing Pipeline: Working towards continuous delivery Deploy Cadence / Risk: More frequent deliveries, lower risk Tightly Coupled Improvement Plan Extract independent domain services with well-defined APIs Consider wrapping infrequently changed, poorly tested components in APIs Loosely Coupled Architecture - Goal Loosely Coupled Architecture Characteristics Components delivered independently Reduced complexity Improved quality feedback loops Relies on clean team … --- ## Retrospectives URL: https://dojoconsortium.org/docs/workflow-management/retrospective/ Description: Regular team practice for inspecting and adapting how work gets done, critical for continuous improvement and preventing entropy Retrospectives are critical for teams that are serious about continuous improvement. They allow the team an opportunity to take a moment to inspect and adapt how they work. The importance of this cannot be overstated. Entropy is always at work, so we must choose to change so that change doesn’t choose us. Recommended Practices Successful Retrospectives A successful retrospective has five parts: Go over the mission of the team and the purpose of retrospective. The team owns where they are right now using Key Performance Indicators (KPIs) they’ve agreed on as a team. The team identifies whether experiments they are running are working or not. If an experiment is working, the team works to standardize the changes as part of daily work. If an experiment is not working, the team either adjusts the experiment based on feedback or abandons the experiment to try something else. Both are totally acceptable and expected results. In either case, the learnings should be shared publicly so that anyone in the organization can benefit from them. The team determines whether they are working towards the right goal and whether the experiments they are working on are moving them towards it. If answer to either of the questions is “No.” then the team adjusts as necessary. Open and honest conversation about wins and opportunities throughout. Example Retro Outline Go over the team’s mission statement and the purpose of retrospective (2 min) Go over the team’s Key Performance Indicators and make sure everyone knows where we are (5-10 min) Go over what experiments the team decided to run and what we expected to happen (5 minutes) What did we learn this week? (10-15 minutes) Should we modify any team documents? (2 minutes) What went well this week? (5-10 minutes) What sinks our battleship? (5-10 minutes) Are we working towards the right things? What are we going to try this week? How will we measure it? (10-15 minutes) Organizing Retros There are some … --- ## Story Slicing URL: https://dojoconsortium.org/docs/work-decomposition/story-slicing/ Description: Techniques for splitting large stories into smaller, vertically-sliced deliveries that provide value independently and reduce batch size Story slicing is the activity of taking large stories and splitting them into smaller, more predictable deliveries. This allows the team to deliver higher priority changes more rapidly instead of tying those changes to others that may be of lower relative value. Recommended Practices Stories should be sliced vertically. That is, the story should be aligned such that it fulfills a consumer request without requiring another story being deployed. After slicing, they should still meet the INVEST principle. Example stories: As an hourly associate I want to be able to log my arrival time so that I can be paid correctly. As a consumer of item data, I want to retrieve item information by color so that I can find all red items. Stories should not be sliced along tech stack layer or by activity. If you need to deploy a UI story and a service story to implement a new behavior, you have sliced horizontally. Do not slice by tech stack layer UI “story” Service “story” Database “story” Do not slice by activity Coding “story” Review “story” Testing “story” Tips If you’re unsure if a story can be sliced thinner, look at the acceptance tests from the BDD activity and see if it makes sense to defer some of the tests to a later release. While stories should be sliced vertically, it’s quite possible that multiple developers can work the story with each developer picking up a task that represents a layer of the slice. Minimize hard dependencies in a story. The odds of delivering on time for any activity are 1 in 2^n where n is the number of hard dependencies. --- ## CD Dependencies URL: https://dojoconsortium.org/docs/cd/cd-dependency-tree/ Description: Visual guide to the dependencies and practices that enable Continuous Delivery The practices and capabilities shown below are based on research and industry standards documented at MinimumCD.org Practices. Overview Continuous Delivery is built on a foundation of practices that depend on each other. This dependency tree shows how fundamental practices like Trunk-Based Development, Test-Driven Development, and Behavior-Driven Development support Continuous Integration, which in turn enables Continuous Delivery. For detailed information about each practice, including implementation guides and research backing, visit practices.minimumcd.org. CD Dependency Tree %%{init: {'securityLevel': 'loose', 'theme': 'base', 'themeVariables': { 'primaryColor': '#ff0000'}}}%% graph BT TDD([Test Driven Development.])-->CI BDD([Behavior Driven Development.])-->TDD TBD([Trunk-based Development.])-->CI CI([Continuous Integration.])-->CD([Continuous Delivery.]) 4-->CI 1([dedicated build server])-->4 2([scripted builds])-->1 3([versioned code base])-->2 4([builds are stored]) CI-->4 5([auto-triggered build])-->CI 2-->6([automated tag & versioning]) CI-->7([pipeline with deploy to prod]) 7-->CD 7-->8([build once, auto-deploy anywhere]) 9([scripted config changes])-->CD 10([standard process for all envs])-->CD 11([automatic DB deploys])-->CD 12([zero downtime deploys])-->CD 13([zero-touch continuous deployments])-->CD 14([defined & documented product development process])-->CI 15([definition of done])-->14 16([prioritized work])-->14 17([working agreements])-->14 18([adopt basic Agile methods]) 19([one backlog per team]) 20([remove boundaries between dev, test, & support]) 21([share the pain]) 22([stable teams]) 23([act on build, quality, test, deploy and operational metrics]) 24([common process for all changes]) 25([component ownership]) 26([decentralize decisions]) 27([extended team collaboration]) 28([frequent commits]) … --- ## Contract Testing URL: https://dojoconsortium.org/docs/testing/contract/ Description: Non-deterministic tests that validate test doubles by verifying contract format against live external systems A contract test is used to validate the test doubles used in a network integration test. Contract tests are run against the live external sub-system and exercises the portion of the code that interfaces to the sub-system. Because of this, they are non-deterministic tests and should not break the build, but should trigger work to review why they failed and potentially correct the contract. A contract test validates contract format, not specific data. – Testing Glossary Contract tests have two points of view, Provider and Consumer. Provider Providers are responsible for validating that all API changes are backwards compatible unless otherwise indicated by changing API versions. Every build should validate the API contract to ensure no unexpected changes occur. Consumer Consumers are responsible for validating that they can consume the properties they need (see Postel’s Law) and that no change breaks their ability to consume the defined contract. Recommended Best Practices Provider contract tests are typically implemented as unit tests of the schema and response codes of an interface. As such they should be deterministic and should run on every commit, pull request, and verification of the trunk. Consumer contract tests should avoid testing the behavior of a dependency, but should focus on comparing that the contract double still matches the responses from the dependency. This should be running on a schedule and any failures reviewed for cause. The frequency of the test run should be proportional to the volatility of the interface. When dependencies are tightly aligned, consumer-driven contracts should be used The consuming team writes automated tests with all consumer expectations They publish the tests for the providing team The providing team runs the CDC tests continuously and keeps them green Both teams talk to each other once the CDC tests break Provider Responsibilities: Providers should publish machine-readable documentation of their interface to … --- ## E2E Testing URL: https://dojoconsortium.org/docs/testing/e2e/ Description: Understanding and implementing End-to-End (E2E) testing in software development End-to-end tests validate the entire software system, including its integration with external interfaces. They exercise complete production-like scenarios, typically executed after functional testing. E2E Test Types of E2E Tests Vertical E2E Tests Target features under the control of a single team. Examples: Favoriting an item and persisting across refresh Creating a new saved list and adding items to it Horizontal E2E Tests Span multiple teams. Example: Going from homepage through checkout (involves homepage, item page, cart, and checkout teams) Note Due to their complexity, horizontal tests are unsuitable for blocking release pipelines. Recommended Best Practices E2E tests should be the least used due to their cost in run time and in maintenance required. Focus on happy-path validation of business flows E2E tests can fail for reasons unrelated to the coding issues. Capture the frequency and cause of failures so that efforts can be made to make them more stable. Vertical E2E tests should be maintained by the team at the start of the flow and versioned with the component (UI or service). CD pipelines should be optimized for the rapid recovery of production issues. Therefore, horizontal E2E tests should not be used to block delivery due to their size and relative failure surface area. A team may choose to run vertical E2E in their pipeline to block delivery, but efforts must be made to decrease false positives to make this valuable. Alternate Terms “Integration test” and “end-to-end test” are often used interchangeably. Resources Testing Strategies in a Microservice Architecture: E2E Introduction The Practical Test Pyramid: E2E Tests Google Test Blog: Just Say No to More End-to-End Tests Example Java @Test(priority = 1, dependsOnMethods = { "navigate" }) @Parameters({ "validUserId" }) public void verifyValidUserId(@Optional(TestConstants.userId) String validUserId) throws Exception { // Valid UserId Test // Act … --- ## Spikes URL: https://dojoconsortium.org/docs/work-decomposition/spikes/ Description: Time-boxed explorations (1-3 days) for work items with high uncertainty that cannot be estimated, used sparingly to reduce risk Spikes are an exploration of potential solutions for work or research items that cannot be estimated. They should be time-boxed in short increments (1-3 days). Recommended Practices Since all work has some amount of uncertainty and risk, spikes should be used infrequently when the team has no idea on how to proceed with a work item. They should result in information that can be used to better refine work into something valuable, for some iteration in the future. Spikes should follow a Definition of Done, with acceptance criteria, that can be demoed at the end of its timebox. A spike should have a definite timebox with frequent feedback to the team on what’s been learned so far. It can be tempting to learn everything about the problem and all of the solutions before trying anything, but the best way to learn is to learn using the problem in front of us right now. Batching learning is worse than batching other kinds of work because effective learning requires applying the learning immediately or it’s lost. Tips Use spikes sparingly, only when high uncertainty exists. Spikes should be focused on discovery and experimentation. Stay within the parameters of the spike. Anything else is considered a waste. --- ## Unplanned Work URL: https://dojoconsortium.org/docs/workflow-management/unplanned-work/ Description: Managing interruptions that prevent finishing planned work, balancing necessary changes with risk, uncertainty, and predictability Unplanned work is any interruption that prevents us from finishing something as planned. There are times when unplanned work is necessary and understandable, but you should be wary of increased risk, uncertainty, and reduced predictability. Cost of Delay Work that has not been prioritized is work that has not been planned. When there are competing features, requests, support tickets, etc., it can be difficult to prioritize what should come first. Most of the time, teams prioritize based on what the customer wants, what the stakeholders want, etc. Cost of Delay makes it easier to decide priorities based on value and urgency. How much money are we costing (or saving) the organization if Feature A is delivered over Feature B? Capacity Planning The most common pitfall that keeps teams from delivering work is unrealistic capacity planning. Teams that plan for 100% of their capacity are unable to fit unknowns into their cadence, whether that be unplanned work, spikes, or continuous experimentation and learning. Planned capacity should fall between 60% and 80% of a team’s max capacity. Tips Plan for unplanned work. Pay attention to the patterns that present themselves, and analyze what kind of unplanned work is making it to your team’s backlog. Make work visible, planned and unplanned, and categorize unplanned work based on value and urgency. --- ## Contract Driven Development URL: https://dojoconsortium.org/docs/work-decomposition/contract-driven-development/ Description: Define contracts between dependencies during design to enable asynchronous development using mocks and fakes for parallel work Contract Driven Development is the process of defining the contract changes between two dependencies during design and prior to construction. This allows the provider and consumer to work out how components should interact so that mocks and fakes can be created that allow the components to be developed and delivered asynchronously. Recommended Practices For services, define the expected behavior changes for the affected verbs along with the payload. These should be expressed as contract tests, the unit test of an API, that both provider and consumer can use to validate the integration independently. For more complicated interaction that require something more than simple canned responses, a common repository that represents a fake of the new service or tools like WireMock can be used to virtualize more complex behavior. It’s important that both components are testing the same behaviors. Contract tests should follow Postel’s Law: "Be conservative in what you do, be liberal in what you accept from others". Tips For internal services, define the payload and responses in the developer task along with the expected functional test for that change. For external services, use one of the open source tools that allow recording and replaying responses. Always create contract tests before implementation of behavior. --- ## 24 Capabilities to Drive Improvement URL: https://dojoconsortium.org/docs/cd/dora-recommendations/ Description: Research-backed practices from the State of DevOps reports and DORA metrics “Our research has uncovered 24 key capabilities that drive improvements in software delivery performance in a statistically significant way. Our book details these findings.” Excerpt From: Nicole Forsgren PhD, Jez Humble & Gene Kim. Accelerate Continuous Delivery Capabilities Use version control for all production artifacts Version control is the use of a version control system, such as GitHub or Subversion, for all production artifacts, including application code, application configurations, system configurations, and scripts for automating build and configuration of the environment. Automate your deployment process Deployment automation is the degree to which deployments are fully automated and do not require manual intervention. Implement continuous integration Continuous integration (CI) is the first step towards continuous delivery. This is a development practice where code is regularly checked in, and each check-in triggers a set of quick tests to discover serious regressions, which developers fix immediately. The CI process creates canonical builds and packages that are ultimately deployed and released. Use trunk-based development methods Trunk-based development has been shown to be a predictor of high performance in software development and delivery. It is characterized by fewer than three active branches in a code repository; branches and forks having very short lifetimes (e.g., less than a day) before being merged into trunk; and application teams rarely or never having code lock periods when no one can check in code or do pull requests due to merging conflicts, code freezes, or stabilization phases. Implement test automation Test automation is a practice where software tests are run automatically (not manually) continuously throughout the development process. Effective test suites are reliable—that is, tests find real failures and only pass releasable code. Note that developers should be primarily responsible for creation and maintenance … --- ## Functional Testing URL: https://dojoconsortium.org/docs/testing/functional/ Description: Understanding and implementing Functional Testing in software development Functional testing is a deterministic test that verifies all modules of a sub-system are working together. It avoids integrating with other sub-systems, preferring test doubles instead. Functional Test Overview Functional testing verifies a system’s specification and fundamental requirements systematically and deterministically. It introduces an actor (typically a user or service consumer) and validates the ingress and egress of that actor within specific consumer environments. Key Points Covers broad-spectrum behavioral tests (UI interactions, presentation-logic, business-logic) Side-effects are mocked and don’t cross boundaries outside the system’s control Differs from E2E tests which have no mocks Recommended Best Practices Write tests from the perspective of an “actor” (user interacting with UI or service interacting with API) Avoid real I/O to reduce flakiness and ensure deterministic side-effects Use test doubles when the system under test needs to interact with an out-of-context sub-system Alternate Terms Component test Resources Component Tests Testing Strategies in a Microservice Architecture: Component Testing Introduction Examples 🚧 Under Construction 🚧 Recommended Tooling Platform Tools Android Google Truth/JUnit 5, Android Espresso iOS XCTest, XCUITest Web Testcafe Java BE TestNG, JUnit5 JS/node BE Framework: jest Assertion & Mocking: expect (jest), supertest, nock, apollo Code Coverage: istanbul/nyc --- ## Visualizing Workflow URL: https://dojoconsortium.org/docs/workflow-management/visualizing-workflow/ Description: Making work visible through visual workflow boards to increase ownership, accountability, and stakeholder transparency Making work visible to ourselves, as well as our stakeholders is imperative in our workflow management process. People are visual beings. Workflows give everyone a sense of ownership and accountability. Make use of a Kanban board Kanban boards help you to make work and problems visible and improve workflow efficiency. Kanban boards are a recommended practice for all agile development methods. Kanban signals your availability to do work. When an individual pulls something from the backlog into progress, they are committing to being available to do the work the card represents. With Kanban boards, your team knows who’s working on what, what the status of that work is, and how long that work has been in progress. Building a Kanban Board To make a Kanban board you need to create lanes on your board that represent your team’s workflow. Adding work in progress (WIP) limits to swim-lanes will enhance the visibility of your team’s workflow. The team only works on cards that are in the “Ready to Start” lane and team members always pick from the top. No “Cherry Picking”. The following is a good starting point for most teams. Backlog Ready to Start Development Ready to Review Blocked Done Tips Track everything: Stories, tasks, spikes, etc. Improvement items Training development Extra meetings Work is work, and without visibility to all of the team’s work it’s impossible to identify and reduce the waste created by unexpected work. Bring visibility to dependencies across teams, to help people anticipate what’s headed their way, and prevent delays from unknowns and invisible work. References Making Work Visible - Dominica DeGrandis --- ## Source Management URL: https://dojoconsortium.org/docs/workflow-management/branching/ Description: Trunk-Based Development practices with short-lived branches, frequent integration, and small pull requests to reduce risk Use Trunk Based Development All branches originate from the trunk All branches merge into the trunk Branches, if used, are very short-lived The smaller the PR, the easier it is to identify issues. The smaller the change, the less risk associated with that change. The trunk can always be built and deployed without breaking production. When needed, use techniques such as Branch by Abstraction or feature flags to ensure backward compatibility. The change includes all appropriate automated tests to validate that the change is deliverable. Unit tests Functional test Contract tests etc. Branching vs. Forking Use the right pattern for the right reason. Branches are the primary flow for CI and are critical for allowing the team to have visibility to work in progress that the team is responsible for completing. Forks are how proposed, unplanned changes are made from outside the team to ensure quality control and to reduce confusion from unexpected branches. Use forks for: Contribution from a contributor outside the team to ensure proper quality controls are followed and to prevent cluttering up the team’s repository with external contributions that may be abandoned. Use branches for: All internal work to keep that work visible to the team. Tips Story Slicing helps break development work into more easily consumable, testable chunks. You don’t have to wait for a story/feature to be complete as long as you have tested that won’t break production. Pull requests should be small and should be prioritized over starting any new development. Common Issues Trunk-based development and continuous integration often take workflow adjustments on the team. The main reasons teams struggle with CI are: Test architecture Work that is too big and/or lacks proper refinement Issues with source code ownership (one repo owned by more than one team) Workflow management within the team References Trunk Based Development. Branching by Abstraction. Feature Flags/Toggles. FAQ … --- ## Defining Product Goals URL: https://dojoconsortium.org/docs/work-decomposition/defining-product-goals/ Description: Transform product vision into measurable objectives with clear timeframes and success criteria to align team efforts Product Goals Product goals are a way to turn your vision for your product into easy to understand objectives that can be measured and achieved in a certain amount of time. Increased transparency into product metrics Measurable Outcome: Increased traffic to product page When generating product goals, you need to understand what problem you are solving, who you are solving it for, and how you measure that you achieved the goals. Initiatives Product goals can be broken down into initiatives, that when accomplished, deliver against the product strategy. Provide one view for all product KPIs. Ensure products have appropriate metrics associated with them. Initiatives can then be broken down into epics, stories, tasks, etc. among product teams, with high-level requirements associated. Epics An epic is a complete business feature with outcomes defined before stories are written. Epics should never be open ended buckets of work. I want to be able to review the CI metrics trends of teams who have completed a DevOps Dojo engagement. Tips Product goals need a description and key results needed to achieve them. Initiatives need enough information to help the team understand the expected value, the requirements, measure of success, and the time frame associated to completion. --- ## Test Doubles URL: https://dojoconsortium.org/docs/testing/test-doubles/ Description: Understanding and implementing Test Doubles in software testing Test doubles are used to create fast, independent, deterministic, and reliable tests. They stand in for real components, similar to how stunt doubles are used in movies. Test Double Types of Test Doubles Key Concepts Test Double: Generic term for any production object replacement in testing Dummy: Passed around but never used; fills parameter lists Fake: Has a working implementation, but not suitable for production Stub: Provides canned answers to calls made during the test Spy: A stub that records information about how it was called Mock: Pre-programmed with expectations, forming a specification of expected calls Resources Test Double Patterns TestDouble Example Java @Before public void init() throws Exception { userService = Mockito.spy(userService); ObjectMapper mapper = new ObjectMapper(); spyData = mapper.readValue(new File(TestConstants.DATA_FILE_ROOT + "user_spy.json"), User.class); Mockito.doReturn(spyData).when(userService).getUserInfo(TestConstants.userId); } @Test public void verifySpyUserDetails() throws Exception { User user = userService.getUserInfo(TestConstants.userId); verify(userService).getUserInfo(TestConstants.userId); verify(userService, times(1)).getUserInfo(TestConstants.userId); Assert.assertEquals(spyData.getManager(), user.getManager()); Assert.assertEquals(spyData.getVp(), user.getVp()); Assert.assertEquals(spyData.getOrganization(), user.getOrganization()); Assert.assertEquals(spyData.getDirector(), user.getDirector()); } @After public void cleanUp() { reset(userService); } Recommended Frameworks Platform Independent Mocking Frameworks Framework Reasoning JSON-Server Simple, great for scaffolding; Follows REST conventions; Stateful Mountebank Allows for more than just HTTP (multi-protocol); Simple to use and configure; Large language support GraphQL Framework Reasoning GraphQL-Faker Supports proxying existing GraphQL APIs; Simple GraphQL directive-based data mocking; Uses faker.js under the hood GraphQL-Tools Built-in utilities … --- ## Code Review URL: https://dojoconsortium.org/docs/workflow-management/code-review/ Description: Lean code review practices prioritizing small changes, team-wide capability, and fast feedback loops with review as second-highest priority Recommended Practices Small changes allow for faster code review and enhance the feedback loops. Everyone on the team is capable of performing code review. Code reviews are the second highest priority for a team behind blocked issues and ahead of WIP. Tips Automate coding standards instead of reviewing for them. Focus the review on the tests and code readability. The tests should meet the acceptance criteria agreed upon by the team. Keep pull requests small. Look into Work Decomposition for guidance. Use synchronous code review to remove communication delays. As the person being reviewed, remember the 10 Commandments of Code Review Thou shalt not take it personally Thou shalt not marry thy code Thou shalt consider all feedback Thou shalt articulate thy rationale Thou shalt be willing to compromise Thou shalt contribute to others’ code reviews Thou shalt treat submitters how thou would like to be treated Thou shalt not be intimidated by the number of comments Thou shalt not repeat the same mistakes Thou shalt embrace the nits References The 10 Commandments of Navigating Code Reviews --- ## Customer Experience Alarms URL: https://dojoconsortium.org/docs/testing/experience-alarms/ Description: Active monitoring that sends requests to test critical customer workflows every minute to ensure system health and catch errors early Customer Experience Alarms are a type of active alarm. It is a piece of software that sends requests to your system much like a user would. We use it to test the happy-path of critical customer workflows. These requests happen every minute (ideally, but can be as long as every 5 minutes). If they fail to work, or fail to run, we emit metrics that cause alerts. We run these in all of our environments, not just production, to ensure that they work and we catch errors early. – Testing Glossary These are different than having log-based alarms because we can’t guarantee that someone is working through all of the golden-path workflows for our system at all times. If we rely entirely on logs, we wouldn’t know if the golden workflows are accurate when we deploy at 3am on a Saturday due to an automated process. These tests have a few important characteristics: They are run in all environments, including production. They aren’t generated from UI workflows, but rather from direct API access They ideally run every minute. If they don’t work (in production) they page someone. Even at 3am. Alternate Terms Synthetic Probes (Google) Canary (Amazon, although it doesn’t mean what Canary means here) --- ## Source Ownership URL: https://dojoconsortium.org/docs/workflow-management/source-ownership/ Description: Team ownership model where everyone takes responsibility for code quality, reducing process overhead and improving delivery Delivery and quality are significantly impacted by teams sharing ownership of the source code. This adds process overhead to ensure everyone knows what’s happening in the code and dilutes quality responsibility. Recommended Practices Utilize automated pipelines to help validate that the product remains releasable before and after any code is merged to the trunk. Limit ownership of a repository to a single “Two Pizza Team” that decides what code to merge. Give all developers on the team access to merge code to the trunk. Give read access to everyone else. Use an innersourcing policy so that people outside of the team know how to contribute to your product. Tips Teams looking to create an InnerSourcing policy can start by applying their Definition of Done to any external contributions. No contributions will bypass the team’s quality process. Automated pipelines validate that PRs from internal and external contributors conform to quality standards. All team members have access to merge to the trunk. InnerSourcing and/or external contributions fork the repository they do not branch. Teams no larger than 10 people, including all roles. References See the CD Common Problems page to learn about team structure problems and many others to avoid in your journey. --- ## Testing Best Practices URL: https://dojoconsortium.org/docs/testing/best-practices/ Description: Comprehensive guide to test-first approaches, naming conventions, test effectiveness, and practices for building maintainable test suites General Recommendation Benefits Gained Use case-centric tests Lower cost to maintain, confidence TDD & BDD Lower cost to maintain, confidence, stability Naming conventions Time to develop, lower cost to maintain Testing your tests Lower cost to maintain, confidence, stability Follow test-type specific recommendations,shifting left on testing Lower cost to maintain, faster speed to execute, less time to develop, confidence, stability Use Case Coverage One of the main points behind testing is to be able to code with confidence. Code coverage is one way developers have traditionally used to represent how confident they feel about working on a given code base. That said, how much confidence is needed will likely vary by team and the type of application being tested. E.g. if working on a life saving med tech piece of software, you probably want all of the confidence in the world. The following discusses how code coverage, if misused, can be misleading and create a false sense of confidence in the code being worked on and as a result, hurt quality. Recommendations on how to manage code coverage in a constructive way will be presented, along with concrete approaches on how to implement them. In simple terms, coverage refers to a measurement of how much of your code is executed while tests are running. As such, it’s entirely possible achieve 100% coverage by running through your code without really testing for anything, which is what opens the door for coverage having the potential of hurting quality if you don’t follow best practices around it. A recommended practice is to look at coverage from the perspective of the set of valid use cases supported by your code. For this, you would follow an approach similar to what follows: Start writing code and writing tests to cover for the use cases you’re supporting with your code. Refine this by going over the tests and making sure valid edge cases and alternative scenarios are covered as well. When done, look … --- ## Average Build Downtime URL: https://dojoconsortium.org/docs/metrics/average-build-downtime/ Description: Time the build stays broken before being fixed - measures team discipline and CI commitment The average length of time between when a build breaks and when it is fixed. What is the intended behavior? Keep the pipelines always deployable by fixing broken builds as rapidly as possible. Broken builds are the highest priority since they prevent production fixes from being deployed in a safe, standard way. How to improve it Refactor to improve testability and modularity. Improve tests to locate problems more rapidly. Decrease the size of the component to reduce complexity. Add automated alerts for broken builds. Ensure the proper team practice is in place to support each other in solving the problem as a team. How to game it Re-build the previous version. Remove tests that are failing. Guardrail Metrics Metrics to use in combination with this metric to prevent unintended consequences. Integration Frequency decreases as additional manual or automated process overhead is added before integration to trunk. --- ## Build Duration URL: https://dojoconsortium.org/docs/metrics/build-duration/ Description: Time for CI pipeline to complete - critical for fast feedback and should be under 10 minutes The time from code commit to production deploy. This is the minimum time changes can be applied to production. This is referenced as “hard lead time” in Accelerate What is the intended behavior? Reduce pipeline duration to improve MTTR and improve test efficiency to give the team more rapid feedback to any issues. Long build cycle times delay quality feedback and create more opportunity for defect penetration. How to improve it Identify areas of the build that can run concurrently. Replace end to end tests in the pipeline with virtual services and move end to end testing to an asynchronous process. Break down large services into smaller sub-domains that are easier and faster to build / test. Add alerts to the pipeline if a maximum duration is exceeded to inform test refactoring priorities. How to game it Reduce the number of tests running or test types executed. Guardrail Metrics Metrics to use in combination with this metric to prevent unintended consequences. Defect rates increase if quality gates are skipped to reduce build time. --- ## Change Fail Rate URL: https://dojoconsortium.org/docs/metrics/change-fail-rate/ Description: Percentage of changes that result in degraded service or require remediation - a key DORA stability metric The percentage of changes that result in negative customer impact, or rollback. changeFailRate = failedChangeCount / changeCount What is the intended behavior? Reduce the percentage of failed changes. How to improve it Release more, smaller changes to make quality steps more effective and reduce the impact of failure. Identify root cause for each failure and improve the automated quality checks. How to game it Deploy fixes without recording the defect. Create defect review meetings and re-classify defects as feature requests. Re-deploy the latest working version to increase deploy count. Guardrail Metrics Metrics to use in combination with this metric to prevent unintended consequences. Delivery frequency can decrease if focus is placed on “zero defect” changes. Defect rates can increase as reduced delivery frequency increases code change batch size and delivery risk. References “Accelerate” Ch2: Measuring Performance - Nicole Forsgren PhD, Jez Humble & Gene Kim --- ## Code Coverage URL: https://dojoconsortium.org/docs/metrics/code-coverage/ Description: Percentage of code exercised by tests - useful indicator but can be gamed, use with caution A measure of the amount of code that is executed by test code. What is the intended behavior? Inform the team of risky or complicated portions of the code that are not sufficiently covered by tests. Care should be taken not to confuse high coverage with good testing. How to improve it Write tests for code that SHOULD be covered but isn’t Refactor the application to improve testability Remove unreachable code Delete pointless tests Refactor tests to test behavior rather than implementation details How to game it Tests are written for code that receives no value from testing. Test code is written without assertions. Tests are written with meaningless assertions. Example: The following test will result in 100% function, branch, and line coverage with no behavior tested. /* Return the sum of two integers */ /* Return null if one of that parms is not an integer */ function addWholeNumbers(a, b) { if (a % 1 === 0 && b % 1 === 0) { return a + b; } else { return null; } } it('Should not return null of both numbers are integers' () => { /* * This call will return 4, which is not null. * Pass */ expect(addWholeNumbers(2, 2)).not.toBe(null); /* * This returns "22" because JS sees a string will helpfully concatenate them. * Pass */ expect(addWholeNumbers(2, '2')).not.toBe(null); /* * The function will never return the JS `NaN` constant * Pass */ expect(addWholeNumbers(1.1, 0)).not.toBe(NaN); }) The following is an example of test code with no assertions. This will also produce 100% code coverage reporting but does not test anything because there are no assertions to cause the test to fail. it('Should not return null if both numbers are integers' () => { addWholeNumbers(2, 2); addWholeNumbers(2, '2'); addWholeNumbers(1.1, 0); }) Guardrail Metrics Test coverage should never be used as a goal or an indicator of application health. Measure outcomes. If testing is poor, the following metrics will show poor results. Defect … --- ## Code Inventory URL: https://dojoconsortium.org/docs/metrics/code-inventory/ Description: Amount of code written but not yet delivered to production - represents unrealized value and risk The lines of code that have been changed but have not been delivered to production. This can be measured at several points in the delivery flow, starting with code not merged to trunk. What is the intended behavior? Reduce the size of individual changes and reduce the duration of branches to improve quality feedback. We also want to eliminate stale branches that represent risk of lost change or merge conflicts that result in additional manual steps that add risk. How to improve it Improve continuous integration behavior where changes are integrated to the trunk and verified multiple times per day. How to game it Use forks to hide changes. Guardrail Metrics Metrics to use in combination with this metric to prevent unintended consequences. Quality can decrease as quality steps are skipped or batch size increases. --- ## Defect Rate URL: https://dojoconsortium.org/docs/metrics/defect-rate/ Description: Measure of escaped defects found in production, indicating test effectiveness and quality processes Defect rates are the total number of defects by severity reported for a period of time. Defect count / Time range What is the intended behavior? Use defect rates and trends to inform improvement of upstream quality processes. Defect rates in production indicate how effective our overall quality process is. Defect rates in lower environments inform us of specific areas where quality process can be improved. The goal is to push detection closer to the developer. How to improve it Track trends over time and identify common issues for the defects Design test design changes that would reduce the time to detect defects. How to game it Mark defects as enhancement requests Don’t track defects Deploy changes that do not modify the application to improve the percentage Guardrail Metrics Metrics to use in combination with this metric to prevent unintended consequences. Delivery frequency is reduced if too much emphasis is place on zero defects. This can be self-defeating as large change batches will contain more defects. --- ## Delivery Frequency URL: https://dojoconsortium.org/docs/metrics/release-frequency/ Description: How often changes are deployed to production - a key DORA metric measuring throughput and team capability How frequently per day the team releases changes to production. What is the intended behavior? Small changes deployed very frequently to exercise the ability to fix production rapidly, reduce MTTR, increase quality, and reduce risk. How to improve it Reduce Development Cycle Time. Remove handoffs to other teams. Remove manual processes. Improve testing and move quality ownership into the team. Move hard dependencies to soft dependencies with feature flags and service virtualization. Focus on Continuous Integration with small changes integrated to the trunk continuously. Use Trunk Based Development to reduce the risk of lost changes and process overhead. How to game it Re-deploying the same artifact repeatedly. Building new artifacts that contain no changes. Guardrail Metrics Metrics to use in combination with this metric to prevent unintended consequences. Change Fail Rate increases as focus shifts to speed instead of quality. Quality decreases if steps are skipped in refining work for the sake of output. --- ## Development Cycle Time URL: https://dojoconsortium.org/docs/metrics/development-cycle-time/ Description: Average time from starting work until release to production - a key flow metric for identifying delivery bottlenecks and improving feedback speed The average time from starting work until release to production. What is the intended behavior? Reduce the time it takes to deliver refined work to production to mitigate the effects of priorities changing and get rapid feedback on quality. How to improve it Decompose work so it can be delivered in smaller increments and by more team members. Identify and remove process waste, handoffs, and delays in the construction process. Improve test design. Automate and standardize the build and deploy pipeline. How to game it Move things to “Done” status that are not in production. Move items directly from “Backlog” to “Done” after deploying to production. Split work into functional tasks that should be considered part of development (development task, testing task, etc.). Guardrail Metrics Metrics to use in combination with this metric to prevent unintended consequences. Quality decreases if quality processes are skipped. Standard deviation of the control chart can show issues being closed too rapidly. References “Accelerate” Ch2: Measuring Performance - Nicole Forsgren PhD, Jez Humble & Gene Kim --- ## Code Integration Frequency URL: https://dojoconsortium.org/docs/metrics/integration-frequency/ Description: How often code is integrated to trunk/main - indicator of CI practice maturity and team collaboration The average number of production-ready pull requests a team closes per day, normalized by the number of developers on the team. On a team with 5 developers, healthy CI practice is at least 5 per day. What is the intended behavior? Increase the frequency of code integration Reduce the size of each change Improve code review processes Remove unneeded processes Improve quality feedback How to improve it Decompose code changes into smaller units to incrementally deliver features. Use BDD to aid functional breakdown. Use TDD to design more modular code that can be integrated more frequently. USe feature flags, branch by abstraction, or other coding techniques to control the release of new features. How to game it Meaningless changes integrated to trunk. Breaking changes integrated to trunk. Guardrail Metrics Metrics to use in combination with this metric to prevent unintended consequences. Quality decreases if testing is skipped. Recommended Practices Trunk Based Development Continuous Integration Feature Flagging --- ## Lead Time URL: https://dojoconsortium.org/docs/metrics/lead-time/ Description: Total time from customer request to delivery in production - measures entire value stream efficiency This shows the average time it takes for a new request to be delivered. This is measured from the creation date to release date for each unit of work and includes Development Cycle Time. What is the intended behavior? Identify over utilized teams, backlogs that need more Product Owner attention, or in conjunction with velocity to help teams optimize their processes. How to improve it Relentlessly remove old items from the backlog. Improve team processes to reduce Development Cycle Time. Use Innersourcing to allow other teams to help when surges of work arrive. Re-assign, carefully, some components to another team to scale delivery. How to game it Requests can be tracked in spreadsheet or other locations and then added to the backlog just before development. This can be identified by decreased customer satisfaction. Reduce feature refining rigour. Guardrail Metrics Metrics to use in combination with this metric to prevent unintended consequences. Quality is reduced if less time is spent refining and defining testable requirements. References InnerSourcing. --- ## Mean Time to Repair (MTTR) URL: https://dojoconsortium.org/docs/metrics/mean-time-to-repair/ Description: Average time to restore service after an incident - a key DORA stability metric measuring recovery capability Mean Time to Repair is the average time between when a incidents is detected and when it is resolved. “Software delivery performance is a combination of three metrics: lead time, release frequency, and MTTR. Change fail rate is not included, though it is highly correlated.” “Accelerate” uses Lead Time for Development Cycle Time. What is the intended behavior? Improve the ability to more rapidly resolve system instability and service outages. How to improve it Make sure the pipeline alway deployable. Keep build cycle time short to allow roll-forward. Implement feature flags for larger feature changes to allow the them to be deactivated without re-deploying. Identify stability issues and prioritize them in the backlog. How to game it Updating support incidents to “closed” before service is restored. Guardrail Metrics Metrics to use in combination with this metric to prevent unintended consequences. Quality decreases if issues re-occur due to lack of improving pipeline quality gates. References “Accelerate” Ch2: Measuring Performance - Nicole Forsgren PhD, Jez Humble & Gene Kim --- ## Quality Metrics URL: https://dojoconsortium.org/docs/metrics/quality/ Description: Comprehensive view of quality indicators including defects, test coverage, and technical debt Quality is measured as the percentage of finished work that is unused, unstable, unavailable, or defective according to the end user. What is the intended behavior? Continuously improve the quality steps in the construction process, reduce the size of delivered change, and increase the speed of feedback from the end user. Improving this cycle improves roadmap decisions. How to improve it Add automated checks to the pipeline to prevent re-occurrence of root causes. Only begin new work with testable acceptance criteria. Accelerate feedback loops at every step to alert to quality, performance, or availability issues. How to game it Log defects as new features Guardrail Metrics Metrics to use in combination with this metric to prevent unintended consequences. [Delivery frequency may be reduced if more manual quality steps are added Build cycle time may increase as additional tests are added to the pipeline Lead time can increase as more time is spent on business analysis --- ## Task Decomposition URL: https://dojoconsortium.org/docs/work-decomposition/task-decomposition/ Description: Breaking stories into smallest independently deployable changes that implement acceptance criteria and maintain flow What does a good task look like? A development task is the smallest independently deployable change to implement acceptance criteria. Recommended Practices Create tasks that are meaningful and take less than two days to complete. Given I have data available for Integration Frequency Then score entry for Integration Frequency will be updated for teams Task: Create Integration Frequency Feature Flag. Task: Add Integration Frequency as Score Entry. Task: Update Score Entry for Integration Frequency. Use Definition of Done as your checklist for completing a development task. Tips If a task includes integration to another dependency, add a simple contract mock to the task so that parallel development of the consumer and provider will result in minimal integration issues. Decomposing stories into tasks allows teams to swarm stories and deliver value faster --- ## Velocity / Throughput URL: https://dojoconsortium.org/docs/metrics/velocity/ Description: Amount of work completed per iteration - team capacity planning metric that should be used carefully, not as productivity measure The average amount of the backlog delivered during a sprint by the team. Used by the product team for planning. There is no such thing as good or bad velocity. This is commonly misunderstood to be a productivity metric. It is not. What is the intended behavior? After a team stabilizes, the standard deviation should be low. This will enable realistic planning of future deliverables based on relative complexity. Find ways to increase this over time by reducing waste, improving planning, and focusing on teamwork. How to improve it Reduce story size so they are easier to understand and more predictable. Minimize hard dependencies. Each hard dependency reduces the odds of on-time delivery by 50%. Swarm stories by decomposing them into tasks that can be executed in parallel so that the team is working as a unit to deliver faster. How to game it Cherry pick easy, low priority items. Increase story points Skip quality steps. Prematurely sign-off work only to have defects reported later. Guardrail Metrics Metrics to use in combination with this metric to prevent unintended consequences. Quality defect ratio goes up as more defects are reported. WIP increases as teams start more work to look more busy. References Harvard Business Review: Six Myths of Product Development Scrum.org: Velocity --- ## Work in Progress (WIP) URL: https://dojoconsortium.org/docs/metrics/work-in-progress/ Description: Count of started but unfinished work - leading indicator of flow problems and context switching Work in Progress (WIP) is the total work that has been started but not completed. This includes all work, defects, tasks, stories, etc. What is the intended behavior? Focus the team on finishing work and delivering it rather than switching between tasks but not finishing them. How to improve it The team should focus on finishing items closest to being ready for production. Prioritize code review over starting new work Prioritize pairing to solve a problem over starting new work Set and do not exceed WIP limits for the team. Total WIP should not exceed team size. Keep the Kanban board visible at all times to monitor WIP How to game it Update incomplete work to “done” before it is delivered to production. Create stories for each step of development instead of for value to be delivered. Do not update work to “in progress” when working on it. --- ## Search Results URL: https://dojoconsortium.org/search/ ---