How to calculate statistical significance (With formulas)
By Indeed Editorial Team
Updated 5 July 2022
Published 30 November 2021
The Indeed Editorial Team comprises a diverse and talented team of writers, researchers and subject matter experts equipped with Indeed's data and insights to deliver useful tips to help guide your career journey.
Statistical significance can help you determine whether a relationship between two circumstances occurs by chance or because of your efforts. Many business professionals use statistical significance to determine whether their data is reliable. Understanding how statistical significance works can help you make informed decisions about project plans and business operations. In this article, we explain how to calculate statistical significance, describe its importance and ways to apply it and share examples of how you can use statistical significance in the workplace.
How to calculate statistical significance
Even professionals without a background can learn how to calculate statistical significance to help them interpret data. Here are 10 steps you can take to calculate statistical significance:
1. Create a null hypothesis
The first step in determining statistical significance is creating a null hypothesis. This involves developing a statement confirming two sets of data do not have any important differences. Essentially, you're confirming a lack of relationship between two variables and stating that one does not affect the others. A null hypothesis is valid even if you disagree with it. For example, you may state that having the newest technology doesn't make a professional more productive at work, even if you believe it does.
2. Create an alternative hypothesis
The next step is creating an alternative hypothesis. This is the opposite of your null hypothesis, so instead of stating no relationship exists between the variables, you state that one influences the other. Again, you may or may not agree with this statement. Using the technology example from the first step, your alternative hypothesis would be that having the newest technology makes a professional more productive at work.
3. Determine the significance level
The significance level, also known as the alpha, refers to the probability that the null hypothesis will compute as false when it's actually true. Typically, significance levels are decimal values that represent percentages. For example, a significance level of 0.05 means there's a 5% chance of determining a difference between the two variables despite there being no differences between them. Data analysts may use different significance levels, but the standard is 0.05.
4. Decide on the type of test to use
There are two types of tests you can use to calculate statistical significance. In a one-tailed test, the critical area of data distribution is one-sided, meaning the test analyses the relationship between the variables in one direction. In a two-tailed test, the critical area of distribution is two-sided, so the test analyses the relationship between the variables in both directions. If the sample lands within the critical area on either test, the alternative hypothesis is true. If it lands in the null area, the null hypothesis is true.
5. Perform a power analysis to find out your sample size
The next step is to perform a power analysis, which allows you to determine the sample size required to produce an accurate measure of statistical significance. To perform a power analysis, determine these four elements:
the type of test you plan to use to calculate statistical significance
the significance level you plan to use for the test
the expected effect size
the sample size
While possible to complete by hand, the process is complex and requires extensive statistical knowledge. Consider using a calculator or software programme to do your power analysis unless you have experience completing them.
6. Calculate the standard deviation
Next, calculate the standard deviation using the following formula:
standard deviation = √((∑|x−μ|^ 2) / (N-1))
∑ = the sum of the data
x = individual data
μ = the data's mean for each group
N = the total sample
Performing this calculation tells you how to spread out your measurements are about the mean or expected value. If you have more than one sample group, standard deviation also helps you determine the variance between the sample groups.
7. Use the standard error formula
Next, use the standard error formula. Presume you have two standard deviations for your two groups. The standard error formula is as follows:
standard error = √((s1/N1) + (s2/N2))
s1 = the standard deviation of your first group
N1 = group one's sample size
s2 = the standard deviation of your second group
N2 = group two's sample size
8. Determine t-score
For the next step, find the t-score. The equation for this is as follows:
t = ((µ1–µ2) / (sd))
t = the t-score
µ1 = group one's average
µ2 = group two's average
sd = standard error
9. Find the degrees of freedom
Next, determine the degrees of freedom. The formula for this is as follows:
degrees of freedom = (s1 + s2) - 2
s1 = samples of group 1
s2 = samples of group 2
10. Use a t-table
For the last step, use a t-table to calculate the statistical significance of your findings. A t-table shows probabilities for different degrees of freedom. You can create a t-table using a spreadsheet, but if you're using a software programme, it may provide you with a t-table based on the data you entered.
The left column contains your p-values, or degrees of freedom. The chart lists your significance levels in ascending order from left to right. To find statistical significance, find the point on the chart where the p-value and significance level meet. The value in that cell is your statistical significance.
The importance of statistical significance
In regards to business, statistical significance is important because it helps you understand that you can attribute the changes you've implemented to various metrics. For example, if you've recently implemented a new application to help your office work more efficiently, statistical significance provides you with the confidence in knowing that it made a positive impact on your company's overall workflow. That is, the app's impact was statistically significant and provided value.
If it turns out the app wasn't statistically significant, this means your business finances and the app are at risk. Make sure to measure the statistical significance for every result to get a more comprehensive calculation and result. To help you make business decisions in the future, consider using business relevance along with statistical significance. This will ensure you do not base your decisions on statistical significance alone.
Ways to use statistical significance
Professionals in many fields use statistical significance to help them interpret data, measure analytics and make important decisions. Sometimes, they collaborate with data analysts to learn more about how this concept applies in complex situations. Here are some instances in which a professional might rely on statistical significance:
Polls and surveys: Companies often ask for feedback from employees or customers, and statistical significance determines the odds that response patterns occurred by chance.
Research experiments: When conducting experiments, researchers may use statistical significance to learn whether the outcome was random or occurred because of their actions.
Projects: Business professionals use statistical significance to measure the relationship between two variables in a project, which can help them develop strategies for current and future projects.
Examples of statistical significance
Reviewing examples of how professionals use statistical significance in the workplace can help you understand how it can benefit you in your role. Here are some examples of calculating statistical significance to evaluate the effectiveness of a project or plan:
Marketing campaign example
Here is an example of calculating statistical significance to determine the effectiveness of a marketing campaign:
You want to attract more customers to your business, so you decide to run an ad campaign. You rely on past ad campaigns to forecast how many print advertisements and digital advertisements you'll need. If you determine that your p-value is above 0.05 or 5%, you'd end up with a result that is not statistically significant. This means that there's a greater than 5% chance that the relationship between the two types of ads might have been by chance. Therefore, this result would indicate that it's not reasonable to use the previous ad campaign as a guide.
Related: 8 essential marketing manager skills
Web design example
Here is an example of using statistical significance to measure the success of a new website design:
You've created a new company website design hoping to attract more customers. You've determined that there was a statistically significant increase in the number of customers since the new website's implementation. Your calculation of the statistical significance resulted in a p-value of 3% or 0.03. Given that it's below 0.05, this is a statistically significant result, meaning that customer increases were not random.
Explore more articles
- 14 social media metrics for tracking campaign success
- Features and benefits of using free online calendars
- What are the pros and cons of having a remote workforce?
- SAFe vs scrum (Plus definition, differences and benefits)
- What is CRM software? (Definition and how to use it)
- 7 practices for success with remote team management
- 7 adaptable leader traits (and how to develop them)
- How to improve online customer service (with steps)
- How to change background colour in Photoshop (with FAQs)
- How to be a socially responsible corporation (with benefits)
- 15 skills for a sales career you can add to your CV
- What is extreme programming? A comprehensive guide