Pilot

01 · Purpose and testable question

Purpose and testable question

A pilot is a bounded operational test of an improvement concept before a wider commitment. It asks whether the proposed mechanism works in practice, whether the process can be operated reliably, and what should change before expansion. It is more informative than a demonstration because it includes ordinary work, exceptions, measurement, and a decision rule. A pilot does not prove a design will work everywhere; its operating scope defines the evidence it can provide.

Use it when there is meaningful uncertainty about effectiveness, usability, implementation, or unintended consequences. Prerequisites include a defined intervention, a reliable baseline or comparator, approved controls, and a plan for stopping or reversing the test. The appropriate duration and sample depend on risk, expected effect, variation, and decision needs. There is no universal minimum number of days or cartons that makes every pilot adequate.

02 · Design comparators and outcomes before starting

Design comparators and outcomes before starting

Fictional Northbank Dispatch tests early carrier assignment with scan confirmation in carrier lane A while lane B continues its current process. The lanes have similar carton mixes and collection schedules, but the team records differences rather than claiming they are identical. It measures both lanes for twenty weekdays before and twenty weekdays during the pilot. Each lane handles 300 cartons per day, giving 6,000 cartons per lane in each period.

The primary outcome is the proportion loaded after the agreed carrier cutoff. Intermediate outcomes are unassigned cartons at staging and carrier-search time. Guardrails are wrong-carrier loading, traceability completeness, confirmation queue time, and aisle obstruction. The team defines inclusion, missing-data handling, and timestamps in advance. It chooses an operationally meaningful improvement target and an acceptable queue range before seeing results. This prevents favorable outcomes being selected after the pilot while unfavorable measures are ignored.

03 · Set operating scope and stop criteria

Set operating scope and stop criteria

The pilot covers normal-size cartons for one carrier lane during staffed dispatch hours. Oversize and hazardous shipments remain outside its scope because they use different handling rules. Operators are trained on normal use, reassignment, unreadable labels, and outage fallback. The team records the configuration version and checks that the comparator does not adopt the same assignment change during the test.

Northbank stops the affected pilot operation immediately for confirmed wrong-carrier loading, loss of traceability, or an unsafe obstruction. It pauses expansion and reviews a sustained queue deterioration against the locally agreed guardrail. These are context-specific criteria approved by the responsible owners, not universal stop rules for all pilots. The reaction plan defines containment and recovery authority. A reversible test still requires dependable controls; learning does not justify exposing customers or staff to a known uncontrolled failure.

04 · Worked outcomes with comparator context

Worked outcomes with comparator context

Before the pilot, lane A has 120 late cartons out of 6,000, or 2.0%; lane B has 108 out of 6,000, or 1.8%. During the pilot, lane A has 60 out of 6,000, or 1.0%; lane B has 102 out of 6,000, or 1.7%. Lane A improves by 1.0 percentage point and lane B by 0.1 percentage point. The simple difference-in-differences estimate is −1.0 − (−0.1) = −0.9 percentage points.

The team also finds no wrong-carrier loads and complete traceability in the observed pilot population. Absence of an event in 6,000 cartons is reassuring within this scope but does not prove its probability is zero. Search time falls while confirmation adds a small step. Operators report that reassignment alerts work on days but are harder to notice at night, prompting a display change before wider testing. The primary outcome and operating evidence inform separate aspects of the decision.

05 · Assess uncertainty and the next decision

Assess uncertainty and the next decision

Under a simplifying independent-binomial calculation, the standard error of the four-proportion difference is about 0.00326, or 0.326 percentage points. An approximate 95% interval is −0.9 ± 0.638 percentage points, approximately −1.54 to −0.26 percentage points. This is illustrative: cartons within a shift or carrier wave may be correlated, and a non-randomized comparator can differ in ways the calculation does not capture. The team therefore examines daily patterns, mix, collection changes, and clustering before making a confidence claim.

Northbank decides the results support a revised pilot on the night shift and another carrier, rather than immediate warehouse-wide rollout. The display issue is corrected and reverified. The team reports the observed effect, uncertainty, exclusions, guardrails, and implementation problems together. It avoids describing a promising estimate as a guaranteed nine-carton-per-thousand saving in every future lane.

06 · Weak versus strong pilots and advanced adaptation

Weak versus strong pilots and advanced adaptation

Weak practice tests only easy work, measures after rather than before, changes several elements without recording versions, or calls a launch party a pilot. Strong practice specifies the question, comparator, outcomes, operating scope, guardrails, and decision in advance. It includes exceptions and documents deviations so the result remains interpretable. A failed pilot can be valuable when it identifies the mechanism or operating condition that needs redesign.

Advanced teams randomize suitable lanes, days, or waves where practical and safe, or use staggered introduction with explicit assumptions. They account for clustering and learning over time, and choose observation length based on the decision’s precision needs. A pilot can separately test technical feasibility, causal effect, and readiness, but the evidence for each should be named. Northbank’s next test asks whether the benefit and controls transfer to a different shift and carrier, making the learning boundary explicit.

07 · Operating conditions and evidence

Operating conditions and evidence

The pilot report should include deviations. Northbank records an unusually late carrier arrival, a temporary staff replacement, a scanner battery failure, and the day the night-shift display changed. Excluding inconvenient days after seeing results would bias the conclusion. If a day is excluded under a pre-specified rule, the report also shows the full data and explains the reason. A design change during the pilot may justify a new phase rather than pooling different versions without comment.

The comparator provides context only if its relationship to the pilot lane is plausible. Lane B may differ in order complexity or benefit indirectly from freed staff in lane A. Northbank checks these spillovers and avoids claiming random assignment where none occurred. The next-wave decision uses the observed improvement, mechanism evidence, guardrails, and transfer risks together. This is stronger than reducing the pilot to one average percentage or a yes/no declaration of success.

08 · Decision record and next step

Decision record and next step

The acceptance discussion distinguishes effectiveness from readiness. Lane A’s lower late-loading proportion supports the proposed effect, while the night-shift display issue shows that operating readiness is incomplete. Both can be true. Northbank therefore avoids an all-purpose success label. It records which question the pilot answered, which question remains open, and which design condition must change before expansion.

The report also states how much precision is needed for the next decision. A modest bounded expansion may be justified with promising evidence and strong recovery controls, while a large capital commitment may require a more precise effect estimate or broader coverage of operating conditions. The required evidence follows the decision’s consequence. Northbank’s next pilot deliberately includes the night shift and a different carrier, because those conditions are where transfer is uncertain. Repeating the same easy lane would add cartons without necessarily resolving the most important remaining question.

Connections

Related methods and phase use

Placed under Improve because of the outcome it supports. Use the method in other phases when its purpose and assumptions fit the work; the phase list is a guide to selection, not a compulsory sequence.

Sources

Professional foundations and further reading

• ASQ: Plan–Do–Check–Act cycle

The explanation and fictional teaching examples are written for LeanTrainingLab. Linked primary sources provide professional foundations; example values and local conventions are stated in the guide. The accepted toolset draws on Six Sigma+Lean Toolset, second edition (Lunau, editor; Springer, 2013). No book text or examples are reproduced here.