Slides for a Presentation on "Non-Psychometric" Survey Design

On August 31st, I gave a new presentation at the Wharton School for University of Pennsylvania staff. I called the presentation, “Some Thoughts on Non-Psychometric Survey Design, Or, Creating Surveys that Measure what you Actually Want to Measure.” By “non-psychometric,” I explained to the attendees, I meant survey design that isn’t going to include validation steps or other psychometrics

Psychometrics is the subfield of Psychology (and especially Personality Psychology) that deals with measure creation.
tools – put differently, survey design for authors who are not interested in making a “validated” measure, but who would still like to take lessons from professional psychometricians.

I’m making my slides and presenter notes available here, with animations and without animations.

The presentation’s abstract is below:

Title: Creating Surveys that Measure what you Actually Want to Measure

People are often poor participants in research. They are inconsistent (e.g., reading questions differently than a survey author intended), complex (e.g., refusing to distill their thoughts about a topic into the one specific aspect a researcher is trying to measure), and biased (e.g., answering either to impress or to spite a researcher).

Unfortunately, people are also often poor authors of research surveys. They write items that are inconsistent (e.g., worded differently throughout a survey in a way that fatigues respondents), complex (e.g., asking multiple questions as part of a single item), and biased (e.g., leading a participant toward a particular response).

This workshop will address the latter in the hope of minimizing the effects of the former.

Topics we will cover:

  • Why people are bad at answering survey questions (impression management, being affected by moment-to-moment circumstances, different people having different baseline responses, and more!)
    • Preventing bias
    • Putting oneself in the shoes of the participant
    • Optimizing vs. satisficing
    • Thinking about test-retest stability
    • Acquiescence (
    • Types of responses (covering topics like this:
    • What does “measuring” psychological traits mean? What is a “valid” measure?
    • What makes for a “good” survey item?
    • Creating surveys using Qualtrics ( (High-level overview of Qualtrics’ interface and functionality)
    • (If we have time and there’s interest) Advanced features of Qualtrics, which could include:
      • Complex survey flows
      • Using Qualtrics’ API (using as an example some code I released several years ago for showing users customized graphs of their responses vs. the average response)