01 · Start from a testable mechanism
Start from a testable mechanism
Design of experiments is a framework for learning by deliberately varying inputs and measuring outcomes. In a warehouse, a hypothesis might be that grouping compatible orders reduces walking but increases sorting errors. Define both the primary response and safeguards before choosing a design. A fast process that damages shipments is not a successful improvement.
Specify controllable factors, feasible levels, nuisance conditions, and the experimental unit. The unit is the smallest entity independently assigned a treatment: perhaps an order wave, not each carton inside it. Identify the objective as comparison, screening, interaction estimation, optimization, or robustness. The objective determines which effects must be estimable and how much uncertainty is acceptable. Reserve resources for confirmation, because the best observed setting may partly reflect noise.
02 · Use randomization, blocking, and replication deliberately
Use randomization, blocking, and replication deliberately
Randomization assigns treatment conditions and, where feasible, run order to prevent systematic alignment with unmeasured influences. It supports a defensible comparison even when every nuisance variable cannot be measured. It does not guarantee perfect balance in a small study. Record the generated order and deviations so the analysis follows the actual assignment.
Blocking groups runs expected to share a nuisance influence, such as day or crew, and compares treatments within those groups. Randomize within blocks rather than calling a fixed sequence blocked. Replication repeats complete experimental units independently and estimates process error. Repeating a stopwatch reading or timing many cartons in one assigned wave is subsampling, not independent replication. Use both where useful, but preserve their different roles in the uncertainty calculation.
03 · Worked randomized blocked pilot
Worked randomized blocked pilot
Suppose a fictional warehouse compares current and redesigned release policies on four days. Each day supplies two comparable independently scheduled waves, one assigned each policy, with the order randomized. Current versus redesigned cycle times are day 1: 20 versus 16; day 2: 22 versus 19; day 3: 18 versus 13; day 4: 21 versus 17 minutes. Paired daily savings are 4, 3, 5, and 4.
The mean saving is four minutes. Squared deviations sum to two, so the standard deviation of daily differences is √(2/3) = 0.8165 and standard error is 0.8165/√4 = 0.4082. A 95% paired interval is 4 ± t0.975,3 × 0.4082 = 4 ± 3.1824 × 0.4082, or 2.7008 to 5.2992 minutes. With four independent day contrasts, uncertainty follows three degrees of freedom, not the number of cartons inside all eight waves.
This pilot supports a positive mean saving under represented conditions. It does not yet show robustness across seasonal peaks or a reduction in errors. Carry the same randomized comparison to additional days and collect error rates with a defined denominator. If a treatment carries over into the next wave, add washout or redesign assignment; ordinary pairing cannot repair contamination.
04 · Select an informative design
Select an informative design
One factor comparisons can estimate a focused difference. Full factorials cross factors and can estimate interactions. Fractional factorials screen more factors with fewer combinations by accepting aliasing. Response surface designs add information for curvature and local optimization. Hard to change settings require a split plot design or another restricted randomization structure rather than pretending the run order was fully randomized.
Choose levels wide enough to reveal operational effects yet within safe feasible limits. Very narrow levels may produce an inconclusive result; extreme impractical levels may answer an irrelevant question. Continuous factors coded −1 and +1 provide comparable scales. Center points can check combined curvature in a two level design, but they do not by themselves identify each factor's separate quadratic effect. A design is useful only if it can estimate the effects the decision actually requires.
05 · Plan the analysis and execution together
Plan the analysis and execution together
Write the intended model before running the study, including factors, interactions, blocks, and the error structure. Ensure the design matrix has sufficient rank and residual information. Estimate power or expected precision using a meaningful minimum effect and credible noise estimate. Balance the cost of additional independent units against the risk of an incorrect decision.
Define the response timing, measurement method, eligibility rules, missing run procedure, and disruption log. Train operators consistently while avoiding coaching that favors one setting. Record actual input levels, not just planned labels. Randomization does not excuse poor measurement. If a run fails, document why and follow a prespecified replacement policy. A replacement chosen because its result looks nicer undermines the comparison.
06 · Judge results rather than selecting a winner blindly
Judge results rather than selecting a winner blindly
Analyze estimated effects with intervals, plots, and model diagnostics. An effect's p value is not its magnitude or savings value. Check whether treatment advantages depend on order mix or nuisance conditions. Inspect run order residuals for drift, and verify equal variance assumptions or use an appropriate model. Repeated testing and post hoc exclusions need transparent exploratory labeling.
Confirm the predicted improvement with new experimental units at the proposed setting. A model selected from the same data has selection uncertainty, and the smallest observed mean is susceptible to winner optimism. Report what was tested, what remains uncertain, and which operating conditions the experiment represented. A result consistent with the mechanism and repeated under different blocks is stronger than a single favorable point in a table.
07 · Strong and weak practice within DMAIC
Strong and weak practice within DMAIC
A strong experiment manipulates a meaningful cause, separates nuisance effects, documents assignment, and measures both customer outcomes and safeguards. A weak experiment changes several inputs without a design, runs all low levels before all high levels, or calls repeated measurements replication. Another weak practice ignores a hard to change factor's shared batch error and produces overly narrow intervals.
DOE often begins in Analyze to verify candidate causes, continues in Improve to select settings, and supplies Control with an operating window. Keep the experiment as part of the causal evidence chain: observed problem, proposed mechanism, manipulated input, measured response, and confirmation. The final handoff should identify the setting, practical effect, uncertainty, conditions, monitoring plan, and circumstances that would trigger a new experiment.
08 · Estimate what additional replication buys
Estimate what additional replication buys
The blocked example has only four independent day contrasts. If the same difference variability persists, sixteen day contrasts would reduce the standard error from about 0.4082 to about 0.2041 minutes, before accounting for the changed t critical value. Collecting sixteen days also broadens operational coverage if those days were sampled appropriately. Repeating more carton measurements on the original four days improves within day measurement but does not produce sixteen independent day effects.
Use this distinction during planning. Reserve runs for uncertain or highly influential conditions and for confirmation at the chosen setting. If budgets force a small exploratory design, state that its role is screening or estimating a rough effect. Do not promise a precise universal conclusion that the run count cannot support. A transparent modest experiment is often more useful than a large dataset whose treatment assignment cannot be reconstructed.
When several responses are measured, identify the primary response and safeguards in advance. Treat additional outcomes as supporting or exploratory unless their comparison family was planned. A cycle time gain and an error rate loss should be evaluated together using customer requirements. Replication and randomization support both claims, but each response may need a different probability model.
Connections
Related methods and phase use
Placed under Analyze because of the outcome it supports. Use the method in other phases when its purpose and assumptions fit the work; the phase list is a guide to selection, not a compulsory sequence.
Sources
Professional foundations and further reading
The explanation and fictional teaching examples are written for LeanTrainingLab. Linked primary sources provide professional foundations; example values and local conventions are stated in the guide. The accepted toolset draws on Six Sigma+Lean Toolset, second edition (Lunau, editor; Springer, 2013). No book text or examples are reproduced here.