Alan Reifman, Texas Tech University
Return to Main Syllabus

Further Issues in Self-Report Measurement

PROBLEM

REMEDY

Intentionally False Responding

  • Denying or inflating reported behavior (donation example; infidelity example)
  • Today Show video of Matt Lauer interviewing Tyra Banks, asking if respondents to her show's survey on teen girls' sexual behavior were "bragging" or "be[ing] sensational"
  • Wanting to look informed (example)

 
Tourangeau & Yan (2007). Sensitive questions in surveys. Psychological Bulletin.

"Respondents in surveys seem to lie for pretty much the same reasons they lie in everyday life --
to avoid embarrassment or possible repercussions from disclosing sensitive information..." (p. 878)

Use one or more of the following to throw out
subjects’ data or make statistical adjustments:
  • Social Desirability Scale, given in addition to your questionnaires of primary interest (see Appendix L of linked document; plus Waring & Reddon, 1983, article on measuring marital intimacy in Journal of Clinical Psychology, via TTU library website )
  • Contradictory answering (example)
  • Direct biochemical/physical verification or simply making respondents think their answers can be independently verified ("bogus pipeline"; example; see p. 42)
  • Bogus items (example)
  • Indirect external confirmation (an unusual example of which)

Note that some of these remedies can pinpoint if
an individual respondent is providing false answers,
whereas others can only detect misreporting for respondents as a group.

Inattentive Responding
  • "Infrequency" items (p. 89)
  • Instruction to leave item blank

Cognitive/Memory Limits
(even when respondent is trying hard)
  • Better prompts/Context reinstatement
    (examples here and here)
  • External confirmation

Acquiescence
(tendency to agree with whatever proposition is before you)
A February 2010 national poll of self-identified Republican voters revealed fairly large percentages taking what many would consider extreme stances.   One analyst suggests that acquiescence may be inflating the percentages.

  • Reverse-score some of the items (ensures
    no high scores due to acquiescence) or use forced-choice format (e.g., who is more qualified to be president, Barack Obama or Sarah Palin?; also ask in reverse order)
Order in Which Questions are Asked (Order Effects)

A humorous example of which is seen in this video clip from the British TV show "Yes, Prime Minister." 

  • Have different versions of the survey (e.g., Form A and Form B), where each form presents the questions in different orders.

Further topics on Pew Research Center's Questionnaire Design page are also worth examining, time-permitting.

Ways of Collecting Self-Report Data and the Trade-offs Involved

Traditional, "Old Fashioned" Approaches: 

Good overview PowerPoint, from University of Illinois, Chicago)

Tourangeau and Yan (2007, p. 878, Psychological Bulletin) note that, "Respondents are less likely to overreport socially desirable behaviors and to underreport socially undesirable ones when the questions are self-administered, when the... bogus pipeline is used, and when the data are collected in private (or at least away from the respondent's parents)." They caution, however, that, "Even when the questions are self-administered, whether by computer or on paper, many respondents still misreport when they answer sensitive questions." 

Newer, Technology-Driven Approaches

"The Evolving World of Survey Research"

"Technological Innovations in Data Collection" (a bit old, but very thorough)

UC San Francisco survey overview document; contains a lot of computer-assisted technologies

Brief overview of computer-assisted interviewing

PDA surveys (here and here)

June 2007 news article on relatively new interview techniques to promote feelings of privacy and honesty of responses, to sensitive topics such as sexual behavior.

Pew Research Center page on Collecting Survey Data discusses cell phone, internet, and other modalities.

Additional Links on Self-Report Measurement
 Question Wording, Online Surveys, Modalities, and Survey Facilities

Dramatic example of question-wording effect ("gay men and lesbians" vs. "homosexuals" in military; February 2010 )

Texas Tech's Earl Survey Research Lab (conducts telephone and mail surveys)

 Interesting commentary about writing survey questions on controversial public issues, and how to interpret results from such polls (with specific reference to the Terri Schiavo controversy in March 2005; i.e., whether the feeding tube of this brain-damaged woman should have been removed and whether the Congress should have gotten involved)

My Health Care Polls website, which among other things, examines the wording of survey items

Article "How to make sense of stories about polls" (discusses question wording, among other things)

Two lists (here and here) of sites that will host web-based surveys (virtually all such hosting sites charge a fee, but often they provide free services for a limited amount of time or limited-size surveys).