On August 31st, I gave a new presentation at the Wharton School for University of Pennsylvania staff. I called the presentation, “Some Thoughts on Non-Psychometric Survey Design, Or, Creating Surveys that Measure what you Actually Want to Measure.” By “non-psychometric,” I explained to the attendees, I meant survey design that isn’t going to include validation steps or other psychometrics
The presentation’s abstract is below:
Title: Creating Surveys that Measure what you Actually Want to Measure
People are often poor participants in research. They are inconsistent (e.g., reading questions differently than a survey author intended), complex (e.g., refusing to distill their thoughts about a topic into the one specific aspect a researcher is trying to measure), and biased (e.g., answering either to impress or to spite a researcher).
Unfortunately, people are also often poor authors of research surveys. They write items that are inconsistent (e.g., worded differently throughout a survey in a way that fatigues respondents), complex (e.g., asking multiple questions as part of a single item), and biased (e.g., leading a participant toward a particular response).
This workshop will address the latter in the hope of minimizing the effects of the former.
Topics we will cover:
- Why people are bad at answering survey questions (impression management, being affected by moment-to-moment circumstances, different people having different baseline responses, and more!)
- Preventing bias
- Putting oneself in the shoes of the participant
- Optimizing vs. satisficing
- Thinking about test-retest stability
- Acquiescence (https://www.qualtrics.com/blog/avoiding-the-yes-bias/)
- Types of responses (covering topics like this: https://www.qualtrics.com/blog/optimal-scale-labels-why-they-matter/)
- What does “measuring” psychological traits mean? What is a “valid” measure?
- What makes for a “good” survey item?
- Creating surveys using Qualtrics (https://upenn.qualtrics.com) (High-level overview of Qualtrics’ interface and functionality)
- (If we have time and there’s interest) Advanced features of Qualtrics, which could include:
- Complex survey flows
- Using Qualtrics’ API (using as an example some code I released several years ago for showing users customized graphs of their responses vs. the average response)