On The Robustness Of Efficient Experimental Designs

Research output: Contribution to conferenceAbstractScientific



Stated Choice (SC) experiments are widely used to acquire understanding on choice behaviour. Nowadays, SC experiments are increasingly being based on so-called “efficient designs”. Efficient designs aim to generate stated choice tasks that maximize the collected information in the data, yielding more reliable parameter estimates with an equal or lower number of observations than traditional orthogonal designs. While earlier research efforts on efficient experimental design mainly focussed on extending the design theory to encompass more advanced models of choice, such as Nested Logit and (Panel) Mixed Logit models, recent efforts are shifting towards understanding the robustness of the modelling results towards the efficient experimental design. These studies focus on exploring to what extent a particular experimental design loses efficiency when the data generating process does not match the model on which the design is based. This literature has predominantly focussed on misspecification in terms of parameter priors, interaction effects and the way in which (correlations between) error terms are modelled (Ferrini and Scarpa 2007; Yu et al. 2008; Bliemer and Rose 2011; Ojeda-Cabral et al. 2016)

However, despite compelling evidence that decision-makers use a wide range of decision rules when making choices (Hess et al. 2012), and despite the rapidly growing interest in the choice modelling community into alternative decision rules (Leong and Hensher 2012; Guevara and Fukushi 2016), robustness issues concerning potential misspecification of the presumed decision rule has attracted only very limited attention within the literature. In fact, to the authors’ knowledge, research on experimental designs has exclusively been based on the (often implicit) assumption that decision-makers make choices based on a (linear-additive) Random Utility Maximization (RUM) rule. As a consequence, it is currently unclear what is the influence of different assumptions regarding the decision rules on the statistical efficiency of the design, and on the performance of different choice models under different design assumptions. For instance, do non-RUM designs differ much from RUM-designs?; does a misspecification of the decision rule result in perhaps highly suboptimal designs?; do RUM designs favour RUM models in terms of model fit?

This paper aims to fill these knowledge gaps. To do so, we construct efficient designs based on a non-RUM model – in casu: a Random Regret Minimization (RRM) model – and assess the effects of the design decision rule[1] analytically as well as empirically. We use an RRM model for our analyses because RRM models are among the more popular non-RUM models. Specifically, we use the P-RRM model (Van Cranenburgh et al. 2015) as this model has very convenient mathematical properties for constructing efficient designs. First, we investigate the effect of decision rule misspecification on the statistical efficiency of the SC design. Specifically, we consider two cases: (1) the case in which the experimental designs are optimized for linear-additive RUM while the true Data Generating Process (DGP) is P-RRM, and (2) the case in which the experimental efficient designs are optimized for P-RRM while the true DGP is linear-additive RUM. After that, we use empirical data (collected specifically for this study) to investigate the influence of the design decision rule on the modelling results.


The methodological contributions of this paper to the experimental design literature are twofold. Firstly, we show that for the P-RRM model (Van Cranenburgh et al. 2015) efficient designs can relatively easily be constructed. Because the P-RRM model has a piecewise linear form, the Asymptotic Variance Covariance matrix – which is needed to construct efficient designs – can be determined analytically. Thereby, we complement the choice modeller’s toolbox for designing efficient experimental designs. Secondly, we extend the experimental design literature by developing new insights on the effects of misspecification of the assumed design decision rule on the statistical efficiency. Finally, the substantive contribution of this paper is that we develop new empirical insights on the robustness of modelling results with respect to the assumed design decision rule.

Key findings

Conventional RUM efficient designs can be statistically highly inefficient in case RRM is the better representation of the actual choice behaviour

Inferences based on empirical SC data on what decision rule (in casu: RUM or RRM) best explains the observed choices are found to be highly sensitive to the particular design that is being used. Model fit differences can be substantial and highly significant.

To the extent that a design is more efficient for one particular decision rule (RUM or RRM), the choice modeller is more likely to conclude – based on comparison of the final Log-Likelihoods – that that particular decision rule is the better representation of the actual observed choice behaviour.
Original languageEnglish
Number of pages1
Publication statusPublished - 2017
EventInternational Choice Modelling Conference 2017 - Vineyard Hotel, Cape Town, South Africa
Duration: 3 Apr 20175 Apr 2017


ConferenceInternational Choice Modelling Conference 2017
Abbreviated titleICMC 2017
Country/TerritorySouth Africa
CityCape Town
Internet address


Dive into the research topics of 'On The Robustness Of Efficient Experimental Designs'. Together they form a unique fingerprint.

Cite this