What Does It Mean to Be Statistically Significant?

Statistics is the art of using knowledge that we're absolutely sure of to make very good guesses about things which we don't know. When something is Statistically Significant, it is so unlikely that it might as well be impossible, or it is so likely that it is practically certain - depending on how you construct your tests.

Let's say that you're talking to your friends after a really hard test, and you want to figure out whether you did better or worse than the rest of your class, on average. In theory, you could find this out by just asking everybody about their score - you could then sum up all the scores and divide by the number of your classmates to get the answer. But there will almost certainly be people who either aren't there or won't tell you - and the teacher is being tight-lipped too. Because of this, you can't know the true average score of the class. You'll need to do a statistical test.

So, you average your friends' scores and you think that the scores in the class are centered around one value with outliers on both sides. A t-test is the perfect choice, and you do the test, setting your hypothesis to be:

$H_0: \mu = y$

and your alternative to be

$H_A: \mu < y$

In other words, you are assuming for the moment that your score is at the mean value, while checking whether it is reasonable that the mean is in fact lower than your score.

You set your $\alpha = .05$, which means that you want there to be a less than 5% chance of the t-test being wrong in order to declare that you have done better than the class average.

You do the test, and your p-value comes out to be $.001$.

This graph is of the number of people with a score, which implies the probability of getting that score.

Your score is beyond the point at which there is a 5% chance of a score that high or higher will occur.

Hooray! Your test is statistically significant!

"Statistically significant" is really just another term for "pretty likely" or, in this case, "pretty unlikely." Your p-value is the probability that you got the results that you did or worse, assuming the null hypothesis $H_0$ is correct. So, there is a 0.1% chance that your test score would be in this high if it were the class mean. Given this, we can be confident in the alternative hypothesis - the class mean grade is very likely to be lower than your grade.

Had you gotten a p-value of .98, for example, you could be pretty confident that your test score is very close to the class mean and not substantially different from it. (98% confident, in fact).

Warning

You must always be careful about "Statistically Significant" results, as they are often less broad than they are stated to be. For example, the result above is dependent on the assumption that the class scores had a roughly bell curved shape. What if the real distribution of scores was not a bell curve, but shaped like this:

Then your test would tell you that you were pretty far ahead of the mean, despite the fact that your score is below it.

Your assumptions must match the truth in order to produce true results. Far worse things can come about than unjustified pride when a professional statistician gets it wrong.

Always think about the assumptions of a study when evaluating its worth. After all, it is possible play with your assumptions to make your data seem significant. If one blindly accepts significance as a sign of truth, one can be manipulated into believing entirely contradictiory things. Many scientists fall into this trap because they need to have "significant" results to be published and get more funds to continue their work.

Try it out yourself with some data about Politics and the US Economy

Statistical Significance

We use statistics to differentiate between strong relationships and random noise in data. We can measure the strength of a relationship by finding the statistical significance.

For instance, we might see that the number of assaults in Dallas goes up as the temperature outside goes up. This is an example of a relationship between two variables: the number of assaults and the temperature outside. As one goes up, the other does, too.

Another word for a relationship between variables is association. So, it's fair for us to say that "there is an association between assaults and outside temperature in Dallas".

Tests for statistical significance tell at the probability that the relationship is due only to random chance. The value used to measure significance is called the $p$-value.

If $p=0.04$, we can say that there's only a $4\%$ likelihood the relationship between the $2$ variables is due to random chance. This suggests strongly that the relationship we see is real.

The most common cut-off for statistical significance is $p=0.05$.

This means that we are only comfortable saying a relationship is significant if there's a $5\%$ chance or less it's due to random chance. A $p$-value higher than that means the association isn't statistically significant.

When are the results of my experiment "statistically significant" according to p-value?

Getting straight to the point,

P-value < $\alpha$→ Reject the null hypothesis (statistically significant)

P-value > $\alpha$→FAIL to reject the null hypothesis (NOT statistically significant )

where $\alpha$ = "alpha" where (1 - $\alpha$) equals the percentage of confidence you want your answer to be within.

####A list of common confidence levels and their corresponding alphas.

What is statistical significance and how do I know if my results are statistically significant?

---

In principle, a statistically significant result is a result that did not occur due to some random chance (usually indicated by a low p-value).

---

When a statistic is significant, it means that you are confident that the statistic is reliable. Statistical significance does not necessarily mean the finding is important or that it gives us some great insight on a world problem.

After conducting a hypothesis test we can test for the significance of our results by comparing our p-value with our chosen confidence level ($\alpha$) "alpha".

• A small p-value less than $\alpha$ (typically less than $\alpha = 0.05$) indicates strong evidence against the null hypothesis, so you would "reject the null hypothesis". When the null hypothesis is rejected, the alternative hypothesis ($H_A$) is said to be statistically significant.

• A large p-value greater than $\alpha$ (typically greater than $\alpha = 0.05$) indicates that we do not have sufficient/significant evidence to reject the null hypothesis, so we "fail to reject the null hypothesis".

---

Note: There is no such thing as "accepting the null hypothesis" in hypothesis testing.

---

Central Tendency: Statistical Significance

Oftentimes, when we are working with variables, their relationships, and statistics, we want to understand whether a relationship between two variables is due to random chance or not. If a relationship is due to random chance, then we can't truly establish a relationship. If a relationship is not due to random chance, then there is a relationship between two variables.

The percent chance that two variables are not related, but are connected due to random chance is known as statistical significance. We use hypothesis tests to determine statistical significance. In general, a sample size of $30$ is a minimum to get an accurate measure of statistical significance. The goal here is to understand the impact of our scientific research. The second arm of impact is effect size.

For example, say we are trying to understand the relationship between the number of ice cream sales and the number of kids who play tennis. We know that as the number of ice cream sales increase, so do the number of kids of play tennis. We discover that the statistical significance is $p=0.3$.

That means that there is a $3\%$ chance that the relationship between the two variables, the number of ice cream sales and the number of kids who play tennis, is only $3\%$! I think we can say with pretty good certainty that the relationship is true.

Looking Further

In later lessons, you will learn about confidence intervals and null vs. alternative hypotheses. We will also cover Type I and Type II errors.

Statistical Significance

by Data Demystified

This video by Data Demystified goes over statistical significance and $p$-values.

Summary

In science, we can never truly prove anything to be true. Instead, science focuses on proving that something isn't true.

Null Hypothesis

A statistically significant result compares a statement to a null hypothesis. A null hypothesis is basically when we say that something doesn't work or when something isn't true. If we were to compare a null hypothesis to the criminal justice system, we could say that a null hypothesis is like the concept of "innocent until proven guilty".

$$\text{null hypothesis = doesn't work}$$

When someone is presenting a statistically significant result, they are saying that there is enough evidence to reject a null hypothesis.

$p$-value

$p$-values indicate the percent likelihood that our results are due to random chance.

The smaller the $p$-value, the greater the confidence in our results. You could also think of it as: the more tests you do that support your hypothesis, the lower your $p$-value.