Multilevel analysis when the number of clusters is small
01 February 2019
A small number of top-level units in multilevel analysis is a problem that has worried comparativists for quite some time. In a paper that is forthcoming in the British Journal of Political Science (Elff, Schaeffer, Heisig, and Shikano 2020 (forthcoming)) my co-authors and I show that this problem can be satisfactorily addressed by only moderate modifications of common techniques of statistical inference: using restricted maximum likelihood (REML) instead of (ML) estimators (Patterson, and Thompson 1971) and by avoiding the assumption of asymptotic normality for the sampling distribution of coefficient estimates and assume a t-distribution instead. (Satterthwaite 1941; Kenward, and Roger 1997)
Scholars who work with cross-national comparative surveys, such as the Eurobarometer, European Social Survey, or Comparative Study of Electoral Systems often worry whether multilevel analysis of these data is possible at all. The countries covered by such surveys form the top-level units in such analysis, but their number is usually much smaller than the number of units in a typical survey sample, apparently too small to conduct statistical analysis with confidence. Indeed, methodological literature exists that suggests that inferences drawn from multilevel analysis may suffer from serious problems: Reported standard errors of model coefficients tend to be too small, confidence intervals too short, and statistical hypothesis tests may lead to false discoveries. In a widely cited article that appeared in the American Journal of Political Science, Daniel Stegmüller argues that while frequentist inference suffer from such shortcomings, Bayesian inference does not, allowing to obtain valid results even if the number of top-level is 10 or less (Stegmüller 2013). His simulation studies suggest that estimates of coefficients of contextual variables may even be severly biased. In our article we demonstrate that improvents in inference can also be achieved using non-Bayesian or “frequentist” methods. Further we reproduce an already known proof that coefficient estimates are unbiased in the linear-normal caste if they exist (Kackar, and Harville 1981). The bias in coefficient estimates which Stegmüller finds in his simulation study is attributed to a flaw in its design.
Elff, Martin, Marlin Schaeffer, Jan Paul Heisig, and Susumu Shikano. 2020 (forthcoming). "Multilevel Analysis with Few Clusters: Improving Likelihood-based Methods to Provide Unbiased Estimates and Accurate Inference". British Journal of Political Science (accepted for publication February 2019).
Kackar, Raghu N. and David A. Harville. 1981. "Unbiasedness of Two-stage Estimation and Prediction Procedures for Mixed Linear Models". Communications in Statistics-Theory and Methods 10(13): 1249--1261.
Kenward, Michael G. and James H. Roger. 1997. "Small Sample Inference for Fixed Effects from Restricted Maximum Likelihood". Biometrics 53(3): 983--997.
Patterson, H. D. and R. Thompson. 1971. "Recovery of Inter-Block Information When Block Sizes Are Unequal". Biometrika 58(3): 545--554.
Satterthwaite, Franklin E.. 1941. "Synthesis of variance". Psychometrika 6(5): 309--316.
Stegmüller, Daniel. 2013. "How Many Countries for Multilevel Modeling? A Comparison of Frequentist and Bayesian Approaches". American Journal of Political Science 57(3): 748--761.