Statistical Surveys: Biased Questions

Bias in Questions

For an overview of bias, click here.

One way that bias can pop up in a study is through poorly constructed survey questions. Basically, a biased question is one that is worded in a way that results in one answer being favored over others.

When your goal is to get data that you can use to make inferences about a target population, biased questions get in the way. They can give you information that does NOT represent the true behaviors or beliefs of your subjects.

Keep in mind that biased questions can be spoken or in writing.

Let's look at some types of biased questions.

Notice how the example question above leads us to answer in a particular way. The phrase "responsible parents" definitely makes this question non-neutral.

Even though it looks like an innocent little question, this question is a disaster waiting to happen.

What if a respondent doesn't use the Kleenex brand, but does use tissues? Should they answer "$0$," or should they answer with the number of non-Kleenex tissues they use? Also, what exactly is an "average day"? Is that a day when I'm not sick, or an average day when I'm sick?

The key with questions is to be as clear and neutral as possible.

This question requires your respondent to provide one answer (Yes or No) to two questions.

Your question asks about increased police presence and safer streets. The word and is often a hallmark of double-barreled questions. What if the respondent wants safer streets but doesn't want increased police presence? How are they supposed to answer?

The best way to manage double-barreled items is to simply break them up into individual questions.

(Video) Bias in Survey Questions

by Lamee Storage

Lamee Storage has a great video explaining question bias in a survey and how to avoid it when you write your own questions.

$\large{\text{Summary}}$

Sometimes the wording of your question can make the person answering the question lean towards any specific answer. This is called bias in your survey questions. Lamee Storage goes into each kind of bias that can come up in your questions.

1. Combining two or more issues: If a question has two parts but expects one answer, there is a bias in the question. His example is the question "Should school days be shorter and start earlier?". Somebody's answer to this question may be "yes" to the first part, and "no" to the second part, but they need to pick one answer for the entire question.

2. Using double negatives: Using double negatives can make the question confusing and trick someone into answering in a way they didn't mean to. Questions like "do you not disagree?" can confuse people, while "do you agree?" would be much clearer.

3. Overlapping answer choices: This is similar to combining issues. Answer choices have to be as separate as possible so each person can pick exactly one answer to each question.

4. Using loaded words: Saying things meant to lean towards one answer. His example is with the phrases "pro-life" and "pro-choice". They both put a positive spin on whichever side of the issue you are on. In a similar case, using the words pesticide vs. using the word poison will show your feelings towards the use of chemicals. Even simpler, asking "would you rather watch an exciting movie or read a boring book" shows a preference towards movies.

5. Using leading questions: Saying things like "don't you agree that..." or "wouldn't you say that..." assume that the person answering already agrees with your opinion. His example is "don't you agree that teachers don't get paid enough". This shows that the expect the person answering already thinks that teachers should be paid more.

$$\text{How do I know if a survey question is biased?}$$

Before diving into bias in survey questions, let's begin by defining "bias."

In general bias is defined as a prejudice or judgement in favor of or against something compared to another. However, in statistics a bias refers to a systematic error. The error can be a result of unfair sampling, surveying, or measurement errors.

In this section we are going to focus on bias survey questions.

There are several types of bias survey questions such as, double barreled questions, leading questions, loaded questions, and unclear/ambiguous questions.

Double Barreled Questions

A double barreled question is a question that touches on multiple topics in the question while only allowing the individual answering the question to respond to one portion.

Ex: "Do you think schools should add more art classes and extend the school year?"

a. yes $\quad$ or $\quad$ b. no

This question does not allow people to express their opinion on the two separate topics, instead it asks that people either agree or disagree with the entire statement. This can be an issue because someone may believe that there should be more art classes but does not think the school year should be extended. In cases like this, the individual answering the question can not express that opinion.

This can be solved by separating the question in to two separate questions i.e.

"Do you think schools should add more art classes?"

a. yes $\quad$ or $\quad$ b. no

"Do you think schools should extend their school year?"

a. yes $\quad$ or $\quad$ b. no

Leading Questions

A leading question is a question that makes an assumption about a topic and is meant to lead respondents to a specific answer. Often times these questions make an allegation and then ask for confirmation. These questions are used to lead people to agree with the person asking the questions. They can also be used to make people think of something in a way they hadn't.

The following two questions are examples of leading questions.

"Only crazy cat ladies like cats, right?

• This question leads individuals to agree with that statement.

"Are you worried about the number of teens without jobs?"

• This question makes people believe that the number of unemployed teens is high and it should concern people. This is done without actually providing any information on the unemployment rate of teens.

Loaded Questions

A loaded question is a question that makes an assumption about an individual and is intended to put the individual answering the question on the defense. The question is typically designed to corner an individual.

Here's an example, the question "Did you stop cheating on tests?".

This question makes the assumption that the individual cheats on tests. The phrasing of this question also makes it impossible not to incriminate oneself. If the person says "yes" then they are admitting to have cheated. If the person says no, they are stating that they are still cheating on tests.

Unclear or Ambiguous Questions

A question that is unclear or ambiguous is typically a question that can be interpreted in multiple ways. For example:

"Do you eat lunch regularly?"

• This question can be interpreted differently depending on what the individual considers as "regularly."