About Surveys

By Scott M. Smith Ph.D.

As market researchers, we are dedicated to a careful appraisal of the market we are surveying. But how much time do we dedicate to the appraisal of our own efforts in this area? Metatheory, or the investigation of investigation, is a critical part of being a superior researcher who understands not only what is being measured, but why.

Metatheory is based upon the organization of the concepts being investigated into models that represent relationships. Survey questions are most effective when they focus on concepts that model a process and confidently represent reality on all significant issues.

How, then, do we measure reality? The two most useful measures of reality to a market researcher are Validity and Utility. The first measure, Validity, refers to the accuracy of the survey model in describing and predicting reality. A sales forecasting model that does not forecast sales with reasonable accuracy, for example, is probably worse than no sales forecasting model at all.

The second measure, Utility, is a different issue. Survey development often mirrors mathematical predictive models that are not incomplete, but too complete. Model developers often try to achieve validity, but are led to include so many variables (with correspondingly difficult data collection problems), that the basic structure of the model becomes buried, input data costs become escalated, and confidence in the model is lost.

To create a survey that meets the standards of Validity and Utility is a daunting task, but absolutely necessary to survey success. For further reading on this subject see Scott Smith’s Metatheory and the Escalating Measurement Quality Issues in Online Survey Research. Delve in, it’s good stuff.