Empirical process control: the three Scrum pillars

If you were building a bridge to a design that has been built a thousand times, you would plan it in detail and follow the plan. If you were building something nobody has built, for users who do not yet know what they want, a detailed plan would be a confident guess. Scrum is made for the second kind of work, and empirical process control is its answer to the guessing.

The idea

The Scrum Guide puts it in two sentences. Scrum is founded on empiricism and lean thinking. Empiricism asserts that knowledge comes from experience and making decisions based on what is observed.

The SBOK® Guide's phrasing is close: in Scrum, decisions are made on observation and experimentation rather than detailed upfront planning, and empirical process control aids learning through experimentation when the problem is not well defined or there are no clear solutions.

Both say the same thing. Do a small amount, see what happened, and let that decide what comes next. The opposite approach, a defined process, assumes the work can be understood in advance. When it can, a defined process is efficient. When it cannot, the plan is fiction.

The Guide describes how Scrum applies the idea: an iterative, incremental approach to optimise predictability and to control risk. Predictability in this sense comes from short cycles, each of which produces evidence, and not from a long forecast.

Three pillars

The Guide says Scrum's events work because they implement the pillars of transparency, inspection and adaptation. Each depends on the one before it.

Transparency

The emergent process and work must be visible to those performing the work as well as those receiving the work. Important decisions in Scrum are based on the perceived state of its three formal artifacts, so an artifact with low transparency leads to decisions that diminish value and increase risk. Put simply: if the people who decide cannot see the truth, they will decide on something else.

The SBOK lists where transparency shows up: a vision statement that stakeholders can read, an open Prioritized Product Backlog, a Sprint Backlog, a Scrumboard and burndown chart, and the regular meetings in which progress is stated aloud.

The Guide adds a line to remember: transparency enables inspection, and inspection without transparency is misleading and wasteful.

Inspection

The Scrum artifacts and the progress toward agreed goals must be inspected frequently and diligently, to detect potentially undesirable variances or problems. The Guide notes that Scrum provides cadence for this in its events. Inspection that happens rarely is not inspection, it is an audit. The Daily Scrum looks at progress toward the Sprint Goal every working day, the Sprint Review inspects the outcome, and the Retrospective inspects how the team worked.

Inspection enables adaptation, and the Guide is blunt about the reverse: inspection without adaptation is considered pointless.

Adaptation

If any aspect of the process deviates outside acceptable limits, or if the resulting product is unacceptable, the process or the materials being produced must be adjusted, and as soon as possible to minimise further deviation. The Guide adds that adaptation becomes more difficult when people are not empowered or self-managing, and that a Scrum Team is expected to adapt the moment it learns anything new.

This is the pillar that most often fails in practice. Seeing a problem is cheap. Changing the plan, the scope, the order or the way of working is the hard part, and in many organisations it needs permission the team does not have.

How the events line up

The Guide states the aim for the events directly: they are designed to provoke change.

What each failure looks like

• No transparency. A status report says green while the Sprint is in trouble. A backlog nobody outside the team can read. Decisions are made from what people say, not from what is visible.
• No inspection. Looking only at the end of a long phase. A Sprint Review that is a presentation, with no discussion of what to change.
• No adaptation. The same impediment appears in four retrospectives. The Product Owner's order never changes despite feedback.
• Adaptation without inspection. Changes made on a hunch, with no evidence of the problem they address. Every change looks like progress, and none can be assessed.

The last one is easy to miss. A team that changes its process every Sprint is adapting, but if it never looks at whether the last change helped, it is not practising empiricism.

A worked example

An invented case. A team sees that three Sprints in a row, the last two stories are not finished.

• Transparency. The board shows the stories started on the last two days of each Sprint. No one hides it, and everyone can see the pattern.
• Inspection. At the Retrospective the team reviews what the board shows and finds that the final stories are started late, after the earlier ones are blocked on review.
• Adaptation. They agree to limit work in progress and to review code the day it is written. The change is small and goes into the next Sprint Backlog.
• Inspection again. At the next Retrospective they compare: both late stories are finished. The change stays. Had it not worked, it would be revised.

No part of this needed a consultant. It needed a visible board, a regular look at it and permission to change.

Common misreadings

• "Empirical means no planning." It does not. It means plans are short, tested against reality and revised. Sprint Planning and the Product Goal are both plans.
• "Transparency means telling everyone everything." The requirement is that the work and artifacts are visible to those who perform and receive the work, in a form they can rely on.
• "Inspection means more meetings." The Guide's events are meant to reduce the need for other meetings, and the aim is regularity.
• "Adaptation means changing the goal." Often it means changing the plan so the goal is still met.

The Agile Manifesto states its version of the idea as a value: responding to change over following a plan.

How to practise it

Empiricism is easiest to feel when a plan meets reality. In the simulator, a project plan is made at the start, and then a blocker, a change request and a review each force the team to look at what actually happened and decide what to change.