Strategies for specification search as a cause of bias and inaccuracy of parameter estimates

An important problem in model development, illustrated by MonteCarlo simulation and Bootstrap

Abstract

As we all know, model predictions and estimation results are erroneous. What we do not know is the size of those errors (had we done so, we would have compensated for them). Never the less, professionalism requires that we couple the model results we use, with some kind of measure of their estimated quality. Accuracy thus is a key issue of modelling – not only to obtain, but to properly describe.

The typical tool we would use for description of model accuracy is standard errors of obtained parameter estimates. However, standard errors are only designed to illustrate a smaller part of those model errors that may arise from the complex process of developing a transport model. This paper discusses and investigates some such limitations in relation to the errors that may arise from specification search . Initial analyses, based on a combination of a real data set and simulation tools, will show that there may be considerable inaccuracy and bias caused by systematic factors that are outside the scope of standard error.

Despite the fact that the concrete example, and implicitly much of the discussion, relates to models for discrete choice applied to transport demand, the general conclusions would apply also to a vast range of other types of modelling.