NEJM recently (2012; 367: 1355-1360) requested better analyses of missing data in randomized trial. This recommendation may be expanded to other study designs, especially since the problem with missing responders and missing responses is increasing. This suggestion will not only provide work for statisticians and epidemiologists in the time to come, it will also provide better justified conclusions and most importantly will open up for other types of sensitivity analyses. This will be a most welcome supplement to the P-value tradition. Conclusions should be based upon robustness of findings after critical analyses, rather than just a P-value or a confidence interval.
Handling missing data has been an active research area for several years and good methods are now available in standard statistical computer packages such as SAS and STATA.
Complete case analysis has been the default analytical principle for many years and may work well when the number of missing data is small but even in this case may large statistical models leave few cases with complete data for all included variables. The complete case analysis is based on the assumption that data are missing at random which may be the case if you at random select participants to provide data e.g. because your resources did not allow all to be in the study. In most situations the “missing at random principle” is not a valid assumption.
NEJM do not recommend simple imputation e.g. by replacing the missing data with the last or next recorded value for this variable, or the mean value for all observations, or a special invented value used for all those with missing data. One of the arguments is that this principle reduces variance and distorts the P-values. Of more importance is that this principle may cause bias (Glymour in Oakes LM, Kaufman JS, Methods in Social Epidemiology 2006), but as always there are no rules without exceptions. Replacing missing data with a single valid code may be the right choice if missingness reflects a certain characteristic we need to adjust for.
NEJM favors estimating-equation methods (complete case are weighted by the inverse of an estimate of the probability of being observed) or multiple imputations where expected values are “guessed” by on regression models using estimated data for those missing. If for example smoking data are missing we may estimate what a likely value could be based on what else we know about this person and we may produce a variance for that estimate that will not inflate our P-value and confidence limits.
Replacing missing confounder data is often a good idea in order to obtain better adjustment for the effect measure of interest. Replacing missing data related to the exposure or outcome data can be justified but should be done with great care.
It is also of interest that NEJM recommends use of sensitivity analyses, at least for missing data. If that will be expanded to other sources of bias we would reach more better documentation for what we can learn from this study that just a P-value or a single estimate with a set of confidence limits.
NEJM is the leading journal in clinical science and we may therefore expect other journals to follow. It may be an important move towards better science with better documentation. P values may serve as the main statistics in the perfectly designed trial with complete and accurate data, no loss to follow-up and causal links that fit the condition for the statistical models. The only problem is that such studies hardly exist and what about observational studies trying to capture the population experience?
NEJM presents suggested guidelines and guidelines are ok as long as we avoid principles, criteria or rules. Except for the “be careful” principle we should not limit freedom of research neither for choosing methods nor in making inference.
—Jørn Olsen, Cesar Victora, Neil Pearce, Shah Ibrahim