Starting CD

• 1: Common Blockers
• 2: Pipeline & Application Architecture

1 - Common Blockers

The following are very frequent issues that teams encounter when working to improve the flow of delivery.

Work Breakdown

Stories without testable acceptance criteria

All stories should be defined with declarative and testable acceptance criteria. This reduces the amount of waiting and rework once coding begins and enables a much smoother testing workflow.

Acceptance criteria should define “done” for the story. No behavior other than that specified by the acceptance criteria should be implemented. This ensures we are consistently delivering what was agreed to.

Stories too large

It’s common for teams using two week sprints to have stories that require five to ten days to complete. Large stories hide complexity, uncertainty, and dependencies.

• Stories represent the smallest user observable behavior change.
• To enable rapid feedback, higher quality acceptance criteria, and more predictable delivery, Stories should require no more than two days for a team to deliver.

No definition of “ready”

Teams should have a working agreement about the definition of “ready” for a story or task. Until the team agrees it has the information it needs, no commitments should be made and the story should not be added to the “ready” backlog.

Definition of Ready

- Story
  - Acceptance criteria aligned with the value statement agreed to and understood.
  - Dependencies noted and resolution process for each in place
  - Spikes resolved.

- Sub-task
  - Contract changes documented
  - Component acceptance tests defined

No definition of “Done”

Having an explicit definition of done is important to keeping WIP low and finishing work.

Definition of Done

- Sub-task
  - Acceptance criteria met
  - Automated tests verified
  - Code reviewed
  - Merged to Trunk
  - Demoed to team
  - Deployed to production

- Story
  - PO Demo completed
  - Acceptance criteria met
  - All tasks "Done"
  - Deployed to production

Team Workflow

Assigning tasks for the sprint

Work should always be pulled by the next available team member. Assigning tasks results in each team member working in isolation on a task list instead of the team focusing on delivering the next high value item. It also means that people are less invested in the work other people are doing. New work should be started only after helping others complete work in progress.

Co-dependant releases

Multi-component release trains increase batch size and reduce delivered quality. Teams cannot improve efficiency if they are constantly waiting. Handle dependencies with code, do not manage them with process. If you need a person to coordinate releases, things are seriously broken.

Handoffs to other teams

If the normal flow of work requires waiting on another team then batch sizes increase and quality is reduced. Teams should be organized so they can deliver their work without coordinating outside the team.

Early story refining

As soon as we decide a story has been refined to where we can begin developing it, the information begins to age because we will never fully capture everything we decided on. The longer a story is “ready” before we being working, the less context we retain from the conversation. Warehoused stories age like milk. Limit the inventory and spend more time on delivering current work.

Manual test as a stage gate

In this context, a test is a repeatable, deterministic activity to verify the releasability of the system. There are manual activities related to exploration of edge cases and how usable the application is for the intended consumer, but these are not tests.

There should be no manual validation as a step before we deploy a change. This includes, but is not limited to manual acceptance testing, change advisory boards (CAB), and manual security testing.

Meaningless retrospectives

Retrospectives should be metrics driven. Improvement items should be treated as business features.

Hardening / Testing / Tech Debt Sprints

Just no. These are not real things. Sprints represent work that can be delivered to production.

Moving “resources” on and off teams to meet “demand”

Teams take time to grow, they cannot be “constructed”. Adding or removing anyone from a team lowers the team’s maturity and average problem space expertise. Changing too many people on a team reboots the team.

One delivery per sprint

Sprints are planning increments, not delivery increments. Plan what will be delivered daily during the sprint.

Skipping demo

If the team has nothing to demo, demo that. Never skip demo.

Committing to distant dates

Uncertainty increases with time. Distant deliverables need detailed analysis.

Not committing to dates

Commitments drive delivery. Commit to the next Minimum Viable Feature.

Velocity as a measure of productivity

Velocity is planning metric. “We can typically get this much done in this much time.” It’s an estimate of relative capacity for new work that tends to change over time and these changes don’t necessarily indicate a shift in productivity. It’s also an arbitrary measure that varies wildly between organizations, teams and products. There’s no credible means of translating it into a normalized figure that can be used for meaningful comparison.

By equating velocity with productivity there is created an incentive to optimize velocity at the expense of developing quality software.

CD Anti-Patterns

Work Breakdown

Workflow Management

Teams

Testing Process

2 - Pipeline & Application Architecture

Whenever teams or areas want to improve their ability to deliver, there is a recommended order of operations to ensure the improvement is effective. This value stream improvement journey’s goal is to provide the steps and guide you to good implementation practices.

Prerequisite: Please review the CD Getting Started guide for context.

• 1. Build a Deployment Pipeline
  • Entangled Architecture - Requires Remediation
    • Common Entangled Practices
    • Improvement Plan
  • Tightly Coupled Architecture - Transitional
    • Common Tightly Coupled Practices
    • Improvement Plan
  • Loosely Coupled Architecture - Goal
    • Common Loosely Coupled Practices
• 2. Stabilize the Quality Signal
  • Remediating Test Instability
• 3. Continuous Improvement
• References

1. Build a Deployment Pipeline

Before any meaningful improvement can happen, the first constraint must be cleared. We need to make sure there is a single, automated deployment pipeline to production. Human intervention after the code is integrated should be limited to approving stage gates to trigger automation where needed. A well-architected pipeline will build an artifact once and deploy that artifact to all required test environments for validation and deliver changes safely to production. It will also trigger all of the tests and provide rapid feedback as near the source of failure as possible. This is critical for informing the developer who created the defect so that they have the chance to learn the reasons the defect was created and prevent future occurrences.

Entangled Architecture - Requires Remediation

With an entangled architecture, there is no clear ownership of individual components or their quality. Every team could cause a defect anywhere in the system because they are not working within product boundaries. The pipeline’s quality signal will be delayed compared to better-optimized team architectures. When a defect is found, it will require effort to identify which team created the defect and a multi-team effort to improve the development process to prevent regression. Continuous delivery is difficult with this architecture.

The journey to CD begins with each team executing continuous integration on a team branch and those branches are integrated automatically into a master CI flow daily.

Any breaks in the pipeline should be addressed immediately by the team who owns the branch.

Common Entangled Practices

Team Structure: Feature teams focused on cross-cutting deliverables instead of product ownership and capability expertise.

**Development Process: Long-lived feature branches integrated after features are complete

Branching: Team branches with each team working towards CI on their branch and daily integration of team branches to the trunk that re-runs the team-level tests.

Inverted Test Pyramid: The “ice cream cone testing” anti-pattern is common. However, the teams should be focusing on improving the quality feedback and engineering tests that alert earlier in the build cycle.

Pipeline: Establishing reliable build/deploy automation is a high priority.

Deploy Cadence / Risk: Delivery cadence in this architecture tends to be extended. This in turn leads to large code change delta and high risk.

Improvement Plan

Find the architectural boundaries in the application that can be used to divide sub-systems between the teams to create product teams. This step will realign the teams to a tightly coupled architecture with defined ownership, will improve quality outcomes, and allow them to further decouple the system using the Strangler](https://martinfowler.com/bliki/StranglerFigApplication.html) process

Tightly Coupled Architecture - Transitional

With tightly coupled architecture, changes in one portion of the application can cause unexpected changes in another portion of the application. It’s quite common for even simple changes to take days or weeks of analysis to verify the implications of the change.

Tightly coupled applications have sub-assemblies assigned to product teams along logical application boundaries. This enables each team to establish a quality signal for their components and have the feedback required for improving their quality process. This architecture requires a more complicated integration pipeline to make sure each of the components can be tested individually and as a larger application. Simplifying the pipelines and decoupling the application will result in higher quality with less overhead.

Common Tightly Coupled Practices

Team Structure: Product teams focused on further de-coupling sub-systems

Development Process: Continuous integration. Small, tested changes are applied to the trunk as soon as complete on each product team. In addition, a larger CI pipeline is required to frequently run larger tests on the integrated system, at least once per day.

Branching: Because CI requires frequent updates to the trunk, Trunk-Based Development](https://trunkbaseddevelopment.com) is used for CI.

Developer Driven Testing: The team is responsible for architecting and continuously improving a suite of tests that give rapid feedback on quality issues. The team is also responsible for the outcomes of poor testing, such as L1 support. This is a critical feedback loop for quality improvement.

Pipeline: CI pipeline working to progress to continuous delivery.

Deploy Cadence / Risk: Deliveries can be more frequent. Risk is inversely proportional to delivery frequency.

Improvement Plan

1. As more changes are needed, the team continues extracting independent domain services](https://www.amazon.com/Implementing-Domain-Driven-Design-Vaughn-Vernon/dp/0321834577) with well-defined APIs
2. For infrequently changed portions of the application that are poorly tested, re-writing may result in lost business capabilities. Wrapping these components in an API without re-architecting may be a better solution.

Loosely Coupled Architecture - Goal

With a loosely coupled architecture, components are delivered independently of each other in any sequence. This reduces complexity and improves quality feedback loops. This not only relies on clean separations of teams and sub-assemblies but also on mature testing practices that include the use of virtual services to verify integration.

It’s critical when planning to decompose to smaller services that Domain Driven Design is used to inform service boundaries, value objects, and team ownership. Services should use good micro-service design patterns

Once we have built our production deployment pipeline, the next most critical constraint to address is the trustworthiness of our tests.

Common Loosely Coupled Practices

Team Structure: Product teams maintain independent components with well-defined APIs.

Development Process: Continuous integration. Small, tested changes are applied to the trunk as soon as complete on each product team.

Branching: Because CI requires frequent updates to the trunk, Trunk-Based Development](https://trunkbaseddevelopment.com) is used for CI.

Developer Driven Testing: The team is responsible for architecting and continuously improving a suite of tests that give rapid feedback on quality issues. The team is also responsible for the outcomes of poor testing, such as L1 support. This is a critical feedback loop for quality improvement.

Pipeline: One or more CD pipelines that are independently deployable at any time in any sequence.

Deploy Cadence / Risk: Deliveries can occur on demand or immediately after being verified by the pipeline. Risk is inversely proportional to delivery frequency.

2. Stabilize the Quality Signal

Establishing a production pipeline allows us to evaluate and improve our quality signal. Quality gates should be designed to inform the team of poor quality as close to the source as possible. This goal will be disrupted by unstable tests.

Remediating Test Instability

Unstable test results will create a lack of trust in the test results and encourage bypassing test failure. To correct this:

• Remove flaky tests from the pipeline to ensure that tests in the pipeline are trusted by the team
• Identify the causes for test instability and take corrective action
  • If the test can be stabilized and provides value, correct it and move it back into the pipeline
  • If it cannot be stabilized but is required, schedule it to run outside the pipeline
  • If not required, remove it

In general, bias should be towards testing enough, but not over-testing. Tracking the duration of the pipeline and enacting a quality gate for maximum pipeline duration (from PR merge to production) is a good way to keep testing efficient.

After stabilizing the quality signal, we can track where most of the defects are detected and the type of defects they are. Start tracking the trends for the number of defects found in each environment and the root cause distribution of the defects to inform the test suite improvement plan. Then focus the improvements on moving the majority of defect detection closer to the developer. The ultimate goal is for most defects to be trapped in the developer’s environment and not leak into the deployment pipeline.

3. Continuous Improvement

After removing noise from the quality signal, we need to find and remove more waste on a continuous basis. We start by mapping the deployment process from coding to production delivery and identifying the choke points that are constraining the entire flow. The process for doing this and the effectiveness are documented in Goldratt’s “Theory of Constraints” (TOC). The TOC states that the entire system is constrained by one constraint and improvement of the system will only be effective once that constraint is resolved.

1. Identify the system constraint.
2. Decide how to exploit the system constraint.
3. Subordinate everything else to the above decisions.
4. Elevate the constraint.
5. If, in the previous steps, a constraint has been broken, go back to step one but do not allow the inertia to cause a system constraint.

Some common constraints are:

• Resource Constraints - resources such as the number of people who can perform the task, access to environments, etc. which block the flow based on its limited capacity for the desired outcomes.
• Policy Constraints - policies, practices or metrics that artificially impede flow due to their poor alignment with the overall performance of the system.

Working daily to relentlessly remove constraints is the most important work a team can do. Doing so means they are constantly testing their improved delivery system by delivering value and constantly improving their ability to do so. Quality, predictability, stability, and speed all improve.

References