01 · Define the decision before testing
Define the decision before testing
A hypothesis test asks how compatible observed data are with a specified null model. It does not identify the most important warehouse problem automatically. Start with a measurable claim, an independent sampling unit, and a decision threshold. For example, does a redesigned picking route reduce mean travel time for comparable orders? Distinguish that scientific question from the investment question of whether the saving exceeds implementation cost.
Write H0, the null hypothesis, and H1, the alternative, before inspecting results. For a paired route comparison, let d be old time minus new time. A two sided test uses H0: μd = 0 against H1: μd ≠ 0. A one sided saving claim uses a directional alternative. Direction must follow the real decision and protocol; changing it after seeing a favorable result inflates the false positive risk.
02 · Match the test to the design
Match the test to the design
Use a paired t test when each old observation has a meaningful new counterpart, such as comparable orders processed by the same picker under a planned crossover. Use a Welch two sample t test for independent continuous groups when equal variance is not justified. ANOVA compares several mean groups. Binary outcomes need methods for proportions or logistic regression. Counts may need Poisson or negative binomial models, and repeated observations on the same truck need a dependence aware analysis.
For paired data, t = (d̄−δ0)/(sd/√n), where d̄ is the mean difference, δ0 is the null difference, sd is its sample standard deviation, and n is the number of independent pairs. Under independent normal differences and a true null, t follows Student's distribution with n−1 degrees of freedom. The raw before and after columns need not each be normal; the distribution relevant to this test is that of the differences.
03 · Worked paired route comparison
Worked paired route comparison
Eight fictional paired orders have old minus new travel times of 2, 1, 3, 2, 0, 2, 1, and 3 minutes. The average saving is 14/8 = 1.75 minutes. The sum of squared deviations from 1.75 is 7.5, so sd = √(7.5/7) = 1.0351. The standard error is 1.0351/√8 = 0.3660 minutes. Against zero, t = 1.75/0.3660 = 4.7819 with seven degrees of freedom.
The two sided p value is approximately 0.00201. At a prespecified α = 0.05, reject the zero mean difference null. A 95% interval for the mean saving is 1.75 ± 2.3646 × 0.3660, or 0.8846 to 2.6154 minutes. Reporting this interval reveals that the data support a positive saving but do not yet demonstrate a saving of at least one minute at the corresponding two sided confidence level.
The numerical result is not proof that the route caused the saving if all old runs happened on congested Mondays and all new runs on quiet Fridays. Randomized order, suitable washout, comparable workload, and complete recording determine the causal strength. Pairing without a defensible pairing rule can make an analysis look more precise without improving its validity.
04 · Read p values and errors correctly
Read p values and errors correctly
A p value is the probability, under the null model and its assumptions, of a test statistic at least as incompatible with that null as the observed statistic. It is not the probability that H0 is true, the probability the result happened by chance, or the size of the operational benefit. A small p value can result from a negligible effect measured very precisely. A large p value can result from a valuable effect measured poorly.
Alpha controls the long run Type I error rate for the prespecified procedure: rejecting a true null. Beta is the probability of failing to reject a specified false null, and power is 1−β. Power depends on the effect size, variability, design, sample size, and test. Failure to reject is not proof of equality. If the decision requires showing two methods are sufficiently similar, define an acceptable equivalence margin and use an equivalence procedure; ordinary nonsignificance answers a different question.
05 · Protect the analysis from hidden choices
Protect the analysis from hidden choices
Set the primary response, contrast, significance level, exclusions, and analysis method before running the study. Testing many zones, responses, days, and subgroups then reporting only the smallest p value produces a misleading impression of evidence. Use planned contrasts or appropriate multiplicity adjustments when a family of claims matters. Exploratory findings are useful but should be labeled and independently confirmed.
Do not repeatedly collect data and stop at the first p value below 0.05 unless the study uses a valid sequential design. Likewise, deleting slow orders because they obscure the result changes the target question unless the exclusion was justified in advance. Investigate outliers as process evidence. If a scanner outage is an operationally relevant part of the process, excluding it may remove the very failure mode the team needs to address.
06 · Check assumptions and choose alternatives
Check assumptions and choose alternatives
Inspect paired differences, run order, clusters, and measurement resolution. Independence comes primarily from study design, not from a normality test. With only eight pairs, a failure to reject normality offers little reassurance. A few extreme differences can dominate both the mean and standard error. A permutation test may be appropriate when treatment labels were randomized under an exchangeability structure; it must follow that actual structure.
A signed rank test is not automatically a distribution free test of the same mean. Its interpretation involves the difference distribution and symmetry assumptions. A sign test focuses on directional tendency and may have lower power. Choose alternatives because they answer the intended question under credible assumptions, not because they deliver a preferable p value. Sensitivity analyses can show whether the conclusion depends on a transformation, unusual pair, or variance assumption.
07 · Judge application quality and DMAIC readiness
Judge application quality and DMAIC readiness
A strong application reports the actual effect, uncertainty, denominator, sampling design, and business implication alongside the p value. It reconciles statistical evidence with process mechanisms and verifies the result on new conditions. A weak application declares every significant factor a root cause, treats p = 0.051 and p = 0.049 as radically different truths, or interprets no significance as no problem.
In Analyze, use tests to challenge specific candidate explanations that emerged from process observation and stratification. In Improve, confirm that manipulating the candidate factor changes the outcome by a useful amount. In Control, maintain the changed process and monitor customer impact. Preserve the test specification and raw observations so later teams can distinguish a planned confirmation from an exploratory search.
08 · Evaluate a useful saving rather than any saving
Evaluate a useful saving rather than any saving
In the worked example, testing zero saving gives strong evidence, yet an investment might require at least one minute. Against δ0 = 1, the statistic is (1.75−1)/0.3660 = 2.0494. A prespecified one sided test asks whether the saving exceeds that practical threshold; it is a different hypothesis from simply detecting a change. Report the actual threshold and inferential choice. Do not reinterpret a two sided result after seeing it. Design the study's precision around the minimum worthwhile saving so the final statistical decision is aligned with the operational one.
Connections
Related methods and phase use
Placed under Analyze because of the outcome it supports. Use the method in other phases when its purpose and assumptions fit the work; the phase list is a guide to selection, not a compulsory sequence.
Sources
Professional foundations and further reading
The explanation and fictional teaching examples are written for LeanTrainingLab. Linked primary sources provide professional foundations; example values and local conventions are stated in the guide. The accepted toolset draws on Six Sigma+Lean Toolset, second edition (Lunau, editor; Springer, 2013). No book text or examples are reproduced here.