A comprehensive study of catch-’em-all measures for evaluating consumer attitudes regarding product features

The purpose of this paper is to examine the effects of rating scale design on the results produced by different designs. The main instigator of this thesis was the result of past research, which demonstrated an apparent bias in the results, which was potentially the result of the applied rating scale system. Therefore, two different rating scale systems were compared, the first being a replication of the rating scale system from the previous research (a combination of two seperate rating scales) and the second being a specifically designed potential improvement over the first. Additionally, the second scale was augmented with a mechanism called Choice- Matching, that aims to induce (more) truthful responses from participants, forming a third and final variant. To test whether the first variant is an improvement over the other and whether Choice- Matching augmentation proves to be an improvement, several different statistical and visual measures where applied, including the Shapiro-Wilk test on normality, Q-Q plots, histograms, kernel density plots, resampling, Pearson’s Chi-Squared test on Goodness of Fit and the independent samples test on proportions. Consequentially, the second rating scale was found to be an improvement over the first. The Choice-Matching augmented scale was burdened with a high drop-off rating, resulting in a significantly lower sample size of finished respondents, compared to the other two variants. Additionally, the drop-off was found to be selective, leading to a significant sample selection bias and a base of respondents that is not representative of the potential consumer base. Therefore, the second rating scale system, unaugmented by the Choice-Matching mechanism, was found to be the preferred variant among the three that were evaluated in this study.