Sampling Bias

In the previous lesson in statistics, we talked about sampling methods, including convenience sampling and random sampling. When conducting research, we want to have a representative sample, which is a sample of a population that is generalizable to the whole target population. We pick a sample of people out of a population, which is considered the sample frame.

Say you have a population of 100 dogs. This population has 50 golden retrievers, 30 poodles, and 20 german shepherds.

$$\text{Population } = \text{50 golden retrievers, 30 poodles, 20 german shepherds}$$

We want to figure out the brand of dog food that all dogs like best! To do that, we need a representative sample of all of the dogs in this population. We have enough dog food for 10 dogs, so we need to use stratified sampling to get a representative sample. We can use ratios to determine what a representative sample would look like.

$$\text{Sample } = \text{5 golden retrievers, 3 poodles, 2 german shepherds}$$

Now consider the possibility that there was some kind of error that prevented us from getting a representative sample. Let's say that $10$ german shepherds were selected to test the dog food because the researchers loved german shepherds. This would be a systematic error in sampling, which would indicate a sampling bias.

If there is some kind of error in sampling, we call that a sampling bias

Sampling Bias

When we are sampling, there may be times there are biases that are not due to the sampling itself, but due to some systematic error in sampling. These biases are called sampling biases. Let's talk through some of the most common types of sampling biases.

Self-Selection: exactly what it sounds like, when members of a population choose to be a part of the sample. Oftentimes, college students are volunteers in psychology research studies for class credit. So, this sample isn't representative because the students are choosing to make up the study sample.

Observer Bias: found primarily in human subject research, this bias occurs when the researcher's preconceptions and assumptions influence the research. For example, researchers are watching a ninth grade math class to determine which student is the best at math. If the researchers have an observer bias of boys being better at math than girls, then that preconception/assumption can influence their decision of who in the math class is best at math.

The biggest issue with sampling bias is our sample is not representative of the population, and so any data or findings can be inaccurate. This can lead to poor internal and external validity.

Internal validity: whether a study is able to accurately represent a cause-and-effect relationship between two variables, ruling out alternative explanations.

External validity: whether a study can be generalized to a larger population.

How to Avoid Sampling Bias

It's not like sampling bias is inevitable in research! Especially in human subject research (i.e., researching people), there are ways that we can avoid sampling bias.

Random sampling: when we randomly select items from a population.

Blinding: when the researcher and/or the participant does not know which study group they've been placed in. Say researchers are testing a new serum to create Super-soldiers. The researchers need to test the serum against a placebo (an injection without the serum). In order to get accurate results, the researchers cannot tell the participants whether they get the serum or the placebo, in case that influences the results. In a double-blinded study, the researchers also would not know whether the participant got the serum or placebo, in order to eliminate observer bias. Blinding will also prevent response bias, where participants may respond the way they think they're supposed to respond.

Having a large enough population can also reduce the effect of sampling bias, also known as the Law of Large Numbers.

Because John and Jenn are only analyzing the data from the teenagers who chose to complete the survey, there is self-selection bias involved.

####Other Points to Note

Whenever we do research, we also must be mindful of survivorship bias, which refers to the issue of a researcher focusing so intently on a sample selected from a population that they overlook those who did not meet the selection criteria. This bias is especially prevalent where there are criteria that participants must meet to be a part of a study. A common example of this bias showing up is during polling for a presidential election.

(Video) Sampling Bias

by the University of Montana CCFWD

This video by the University of Montana CCFWD goes over the main types of sampling bias.

Summary

Because we cannot survey an entire population, we need to use a representative sample to understand our larger population. However, when we are creating this sample, we can run into sampling biases, which is when certain individuals in a population are excluded from the sample because of some systematic error. This would indicate that our random sample is not actually random.

An example is trying to understand how Americans will vote in an upcoming election. Because we cannot survey every single American, we survey a sample of the population. Sampling bias occurs as we finding the people to be a part of our sample.

Say we call people to ask how they will vote. That means we are systematically not surveying people who don't answer the phone or who do not have a phone.

Our sample will be full of people who:

• Answer the phone

• Talk to us on the phone

• Willing to do the survey

It's possible that the people who fit those three criteria have some sort of difference than the people who don't. Hypothetically, the people who do those three things are more likely to vote Democrat than Republican, which will skew our sample toward people who vote Democrat. That may make us think that the Democrat is more likely to win, but that could just be because the sample is not representative of the population.

Sampling Bias

Before we learn more about the types of sampling in statistics, we want to first understand what not to do. That's why we are first going to learn about sampling bias.

Sampling bias happens when a sample does not fairly represent the whole population. Imagine a population made up of four green marbles and 4 orange marbles. This means that half of the population is green and half of the population is orange.

Now imagine that we choose a sample that only contains the top circles: the green circles. The sample makes it look like the whole population is green, even though that is not true. This is an example of a biased sample. The bias happened because the orange circles were left out of the sample.

A better sampling method would give both green and orange circles a fair chance of being chosen. A fair sample is more likely to represent the population accurately.

Avoiding sampling bias helps us to draw better conclusions about a larger group. In the next lesson, you are going to learn about one more type of bias: omitted variable bias. Learning about biases will become important as we learn about the different types of research studies: observational studies, surveys, and experiments.