Scope Plausibility
Scope Plausibility asks whether a study's design aligns with the populations to which it aims to infer. Just as design trumps analysis for internal validity, careful design choices are foundational for external validity. We organize Scope Plausibility into three components that move from specifying targets, to planning samples, to assessing what was actually achieved.
Components of Scope Plausibility
SP1: Target Populations
Researchers must specify eligibility criteria for all STOUT dimensions and ensure target populations correspond to theoretical populations of interest. Rich covariate information helps model and account for potential effect modifiers that drive heterogeneity.
SP2: Planned Samples
Researchers must plan samples so real treatment heterogeneity isn't mistaken for random variation. Stratification by key modifiers is typically the safer default, capturing meaningful differences across groups and reducing spurious patterns.
SP3: Actual Samples
Researchers must assess whether actual samples still support credible inference. Nonresponse, attrition, or unanticipated differences can disrupt even well-planned designs, requiring documentation, adjusted estimands, or statistical corrections.
Design Trumps Analysis
Design plays the same foundational role for external validity that it does for internal validity. While analytical adjustments can help, they cannot fully compensate for a design that fails to capture the populations and variation needed for credible generalization. Planning for external validity from the start—through clear eligibility criteria, stratified sampling, and careful documentation—puts researchers in the strongest position to draw population-level inferences.