Truth in Survey Research

survey

Being surveyed is now a routine part of life for many people, especially students, who might fill out course evaluations, mental health checklists, and questionnaires about their substance use multiple times in a single year. Recall your thoughts the last time you were surveyed about something personal: Why are they asking me this? What are they going to do with this information? Why would I tell you? If these questions sound familiar, we wouldn’t be surprised if some of your responses were carefully crafted to be strategic rather than strictly truthful. In our current research, Melissa Wolf and I acknowledge how big a problem lying on school surveys can be (huge) and investigate why students do it. By better understanding the motivations of survey lying we hope to help researchers prevent and detect it. We argue that without a good idea about how much lying is going on, and what kind of lying, much school-based statistical research is vulnerable to distortion.

Finding the Liars

To study the phenomenon of survey lying (technically called “invalid response”) we used a procedure known as cognitive interviewing, a guided technique that helps people explain their mental processes in high detail. The stimuli we asked students to consider were based on a popular school mental health survey and a section of the California Healthy Kids Survey that included sensitive items pertaining to harassment, bullying, sexual orientation, and drug and alcohol use on campus. We started by asking students to identify items that were difficult for them to answer, and before long we were always talking about lying.

How Do You Lie?

Using this method, we found evidence of four basic types of invalid response behavior: mischievous responses, low-effort responses, reticent responses, and subversive responses. The first two types of responses were already well-known to researchers, while the last two were not previously identified. Mischievous response happens when people don’t take the research situation seriously, often reporting a highly unusual combination of demographics that they find amusing (e.g. they are blind, eight feet tall, orphaned, and in a gang) and then giving extreme responses to survey items (e.g. high rates of depression, high rates of drug use). Low-effort response happens when people are not invested in the survey and have little interest in giving thoughtful responses (e.g. choosing the “neutral” option on every question).

The two types of invalid response we introduce to the survey literature are reticent and subversive response. Reticent response is the withholding of truthful responses because of the personally sensitive nature of items. Examples of reticent responders may include suicidal teens who don’t want people to know that they are suicidal or students who don’t want to confess to being addicted to opiates. Subversive response happens when participants speculate about intended uses or interpretations of the survey and give misleading responses that they believe will bring about desirable results. For example, a student who wants more services for people at her school may untruthfully indicate that she needs these services on an anonymous survey. Individual students reported that they might use several of these response styles in a single survey.

Final Thoughts

Survey lying is common and there are a variety of motivations for doing it – so where do we go from here? Schools, we argue, should work towards earning the institutional trust of students and try to present tangible outputs from survey research. As one of our participants put it, “I would love to take [this survey] if it was like ‘OK, we’re going to look at these results and cater to the students… what do you guys need?’” For their part, academics should clearly explain to participants any beneficial outcomes of survey research and be clear about the way that confidential information will be handled. In our future work on survey lying we will continue to develop methods that researchers can use to identify “hot” items that attract a lot of lying and to adjust their statistical confidence about those items accordingly. We also plan to test some of our hypotheses about the relationship between institutional trust, social marginalization, and telling the truth.

About the Author

Anthony Clairmont is a fourth year graduate student in the Gevirtz Graduate School of Education at the University of California Santa Barbara. His research interests include measurement, mixed qualitative and quantitative methods, and the language and culture in schools. He can be contacted at clairmont@ucsb.edu. 

 

 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s