Geert Van Kollenburg

This dissertation focusses on testing model fit of the latent class (LC) model. This model is a powerful tool with which observations can be clustered based on their combination of responses to a number of categorical variables. The research leading to this dissertation focused on testing whether a particular model can correctly describe the observed data by means of p values. In the different chapters p values are compared, improved, and sped up, respectively.

In the second chapter, the frequentist properties (type I error rates and power) of the most common p values were evaluated for many commonly used statistics in LC analysis. A simulation study was conducted which involved calculating different p values for LC models based on sparse and non-sparse contingency tables. It was found that the use of asymptotic p values resulted in very distorted type I error rates when contingency tables were sparse. The parametric bootstrap and model-based PPC performed much better in this regard than the asymptotic method. It was also found that a number of statistics which used second order marginals of the tables were not strongly influenced by sparseness. The power study suggested that nearly all combinations of p value and statistics can be used detect misfit when the number of LCs is misspecified. When sample sizes become very small, however, we should resort to the local fit measures.

The third chapter discusses the distribution of the posterior predictive p value (ppp). The common interpretation of a p value, requires that it is uniformly distributed if the null model is true. Since this uniformity assumption is not true for the ppp, a posterior-calibrated ppp was proposed which is obtained by calibrating the ppp under the posterior given the observed data under the null model. The advantage of this posterior-cppp is that it has all the advantages of the original ppp, but results in nominal type I error rates and has a (much) higher power to detect model misfit. The benefits of the calibrated p value over the original ppp were demonstrated in a number of simulation studies and empirical applications with different testing problems for independence models, linear regression models and latent class models. All studies showed that the proposed posterior-calibrated ppp was uniformly distributed under the null and had much higher power to detect model misfit than the original ppp.

The methodology proposed in the fourth chapter improves on existing methods by eliminating the need for repeated model estimation, while still only requiring maximum likelihood estimates. In this sense it is a best-of-both-worlds approach by combining the flexibility of Bayesian tests with frequentist estimation procedures. The idea is to directly compare observed data with model-generated data in a way that requires no time-consuming model estimation procedures. This comparison can be based on specific characteristics corresponding to those aspects of the data (e.g., descriptive statistics) that the researcher finds most important. If the observed data differ consistently from the model-generated data, we have strong evidence that the model under consideration could not have produced the observed data. The conducted Monte Carlo study showed that the method is much faster than traditional resampling techniques, though it provides a more conservative test. Unsurprisingly, the more information about the data a statistic incorporates, the higher its power to indicate model misfit in case of non-fitting models. An empirical example showed that the new method led the same conclusions as the parametric bootstrap. It is noteworthy to state that the new method was computationally three orders of magnitude faster than the bootstrap.