Analysis of Variance (ANOVA)

ANOVA is a statistical technique used to compare the means of three or more groups, widely applied in fields like agriculture, biology, psychology, and social sciences to analyze experimental data and assess the effects of different treatments or factors on a response variable. This tutorial will guide you through performing ANOVA in Python.

Introduction

Analysis of Variance (ANOVA) is a statistical technique used to compare the means of three or more groups, enabling us to determine whether there are significant differences between them. This method is widely used in research to help us understand whether the differences we observe among group means are due to genuine differences in the populations or simply random sampling variation.

ANOVA works by breaking down the total variance observed in the data into different components: the variance between group means and the variance within each group. It then examines whether the between-group variance is significantly larger than the within-group variance. If it is, this indicates that there are real differences among the groups.

There are various types of ANOVA, including one-way ANOVA, which compares the means of three or more groups on a single independent variable, and two-way ANOVA, which examines the influence of two independent variables. More complex designs, such as factorial ANOVA, can be used for multiple factors.

During ANOVA, an F-statistic is generated, which compares the ratio of the between-group variance to the within-group variance. If the calculated F-value exceeds a critical value based on the chosen significance level, usually 0.05, it suggests that there is a statistically significant difference between at least two group means.

Post-hoc tests, such as Tukey’s HSD or Bonferroni correction, are often used after ANOVA to pinpoint which specific groups differ from each other if the overall ANOVA result is significant. It’s important to note that ANOVA makes certain assumptions, such as the data following a normal distribution and the groups having equal variances. Violations of these assumptions may affect the reliability of the ANOVA results.

Overall, ANOVA is a powerful tool for comparing means across multiple groups simultaneously. It’s commonly used in various fields, such as psychology, biology, economics, and more to analyze experimental or observational data. By comparing group means, ANOVA helps us to understand the differences between groups and identify significant patterns in the data.

Analysis of Variance (ANOVA) is a statistical method to compare the means of three or more groups to determine if there are significant differences among them. It decomposes the total variance in the data into:

  1. Between-Group Variability (SSB): Variance due to differences between group means.
  2. Within-Group Variability (SSW): Variance due to individual observations within groups.

The key statistic in ANOVA is the F-statistic, which compares the ratio of between-group variance to within-group variance. A large F-value suggests significant differences between at least two group means. The formulas are:

  1. Sum of Squares Between (SSB): \[ SSB = \sum_{i=1}^k n_i (\bar{x}_i - \bar{x}_{\text{grand}})^2 \] where \(n_i\) is the sample size of group \(i\), \(\bar{x}_i\) is the mean of group \(i\), and \(\bar{x}_{\text{grand}}\) is the overall mean.

  2. Sum of Squares Within (SSW): \[ SSW = \sum_{i=1}^k \sum_{j=1}^{n_i} (x_{ij} - \bar{x}_i)^2 \]

  3. Degrees of Freedom:

    • Between groups: \(df_{\text{between}} = k - 1\)
    • Within groups: \(df_{\text{within}} = N - k\)

  4. Mean Squares:

    • \(MS_{\text{between}} = SSB / df_{\text{between}}\)
    • \(MS_{\text{within}} = SSW / df_{\text{within}}\)

  5. F-Statistic: \[ F = \frac{MS_{\text{between}}}{MS_{\text{within}}} \]

If the F-value exceeds a critical value (based on a significance level, typically 0.05), it indicates significant differences. Post-hoc tests like Tukey’s HSD or Bonferroni correction can identify which groups differ. ANOVA assumes: - Independence: Observations are independent across groups. - Normality: Data in each group is approximately normally distributed. - Homogeneity of variances: Equal variances across groups.

ANOVA from Scratch

Generate Synthetic Data

We’ll create synthetic data with three groups using NumPy.

Code 
import numpy as np
import pandas as pd

# Set seed for reproducibility
np.random.seed(123)

# Create synthetic data with 3 groups
group1 = np.random.normal(loc=5, scale=1.5, size=10)  # Group 1: mean=5
group2 = np.random.normal(loc=6, scale=1.5, size=10)  # Group 2: mean=6
group3 = np.random.normal(loc=7, scale=1.5, size=10)  # Group 3: mean=7

# Combine into a DataFrame
data = pd.DataFrame({
    'weight': np.concatenate([group1, group2, group3]),
    'group': np.repeat(['Group1', 'Group2', 'Group3'], 10)
})

# View the first few rows
print(data.head())

Perform Manual ANOVA

We’ll compute the ANOVA components manually.

Code 
# Extract response and group
response = data['weight']
groups = [data[data['group'] == g]['weight'] for g in data['group'].unique()]

# Compute grand mean and group statistics
grand_mean = np.mean(response)
k = len(groups)  # Number of groups (3)
n_total = len(response)  # Total sample size (30)

# Calculate SSB (Between-Group Sum of Squares)
group_means = [np.mean(g) for g in groups]
group_n = [len(g) for g in groups]
ssb = sum(n * (m - grand_mean)**2 for n, m in zip(group_n, group_means))

# Calculate SSW (Within-Group Sum of Squares)
ssw = sum(sum((x - np.mean(g))**2 for x in g) for g in groups)

# Degrees of Freedom
df_between = k - 1  # 3 - 1 = 2
df_within = n_total - k  # 30 - 3 = 27

# Mean Squares
ms_between = ssb / df_between
ms_within = ssw / df_within

# F-Statistic and P-Value
from scipy.stats import f
f_value = ms_between / ms_within
p_value = f.sf(f_value, df_between, df_within)

# Assemble ANOVA Table
anova_table = pd.DataFrame({
    'Source': ['Between Groups', 'Within Groups'],
    'SS': [ssb, ssw],
    'df': [df_between, df_within],
    'MS': [ms_between, ms_within],
    'F': [f_value, np.nan],
    'p-value': [p_value, np.nan]
})
print(anova_table)

Validation with Built-in Functions

We can validate using statsmodels.

Code 
from statsmodels.formula.api import ols
from statsmodels.stats.anova import anova_lm

# Perform one-way ANOVA
model = ols('weight ~ group', data=data).fit()
anova_results = anova_lm(model)
print(anova_results)

ANOVA with Python PackagesS

We’ll use statsmodels, pingouin, and seaborn for ANOVA and visualization.

Code 
import importlib.util
import sys

# List of required packages
packages = ['numpy', 'pandas', 'statsmodels', 'scikit-learn', 'matplotlib', 'seaborn', 'pingouin']

# Check and install missing packages
# Check and install missing packages
for package in packages:
    if not importlib.util.find_spec(package):
        try:
            import pip
            pip.main(['install', package])
        except ImportError:
            print(f"Failed to install {package}. Pip is not available.")

Install and Import Packages

Code 
# Import libraries
import statsmodels.api as sm
from statsmodels.formula.api import ols
import pingouin as pg
import seaborn as sns
import matplotlib.pyplot as plt

Generate Experimental Data

Create a dataset with four treatments, six varieties, and four replications.

Code 
# Set parameters
np.random.seed(1256)
treatments = 4
replications = 4
varieties = 6
n = 1

# Create empty DataFrame
exp_df = pd.DataFrame({
    'Treatment': [],
    'Replication': [],
    'Variety': [],
    'Yield': []
})

# Generate random data
for t in range(1, treatments + 1):
    for r in range(1, replications + 1):
        for v in range(1, varieties + 1):
            obs = np.random.normal(loc=3*t + 0.95*r + 0.90*v, scale=0.25, size=n)
            temp_df = pd.DataFrame({
                'Treatment': [f'T{t}'] * n,
                'Replication': [r] * n,
                'Variety': [f'V{v}'] * n,
                'Yield': obs
            })
            exp_df = pd.concat([exp_df, temp_df], ignore_index=True)

# Print first 10 rows
print(exp_df.head(10))

One-way ANOVA

One-way ANOVA tests for differences in means across groups based on one independent variable.

Code 
# Perform one-way ANOVA
anova_one = pg.anova(data=exp_df, dv='Yield', between='Treatment')
print(anova_one)

Two-way ANOVA

Two-way ANOVA examines the effects of two independent variables and their interaction.

Code 
# Perform two-way ANOVA
anova_two = pg.anova(data=exp_df, dv='Yield', between=['Treatment', 'Variety'])
print(anova_two)

Multiple Comparisons

Post-hoc tests identify which specific groups differ after a significant ANOVA result.

Tukey’s HSD Test

Code 
# Tukey's HSD for one-way ANOVA
tukey_one = pg.pairwise_tukey(data=exp_df, dv='Yield', between='Treatment')
print(tukey_one)

# Tukey's HSD for two-way ANOVA (Treatment:Variety interaction)

# Create an interaction column
exp_df['Treatment_Variety'] = exp_df['Treatment'] + '_' + exp_df['Variety']

# Perform Tukey's HSD for the interaction
tukey_two = pg.pairwise_tukey(data=exp_df, dv='Yield', between='Treatment_Variety')
print(tukey_two.head(20))

Pairwise t-tests with Bonferroni Correction

Code 
# Pairwise t-tests with Bonferroni correction
pairwise_t = pg.pairwise_tests(data=exp_df, dv='Yield', between='Treatment', padjust='bonf')
print(pairwise_t)

Box/Violin Plots

Visualize group differences with boxplots.

Code 
# Boxplot for Treatment
plt.figure(figsize=(7.5, 5.5))
sns.boxplot(x='Treatment', y='Yield', data=exp_df)
plt.title('Main Effect of Treatment', fontsize=14)
plt.ylabel('Yield (t/ha)')
plt.xlabel('Treatment')
plt.show()

# Boxplot for Variety
plt.figure(figsize=(6.5, 4.5))
sns.boxplot(x='Variety', y='Yield', data=exp_df)
plt.title('Main Effect of Variety', fontsize=14)
plt.ylabel('Yield (t/ha)')
plt.xlabel('Variety')
plt.show()

Two-way Repeated Measures ANOVA

For designs with repeated measures, we use pingouin’s rm_anova.

Code 
# Two-way repeated measures ANOVA
rm_anova = pg.rm_anova(data=exp_df, dv='Yield', within=['Treatment', 'Variety'], subject='Replication')
print(rm_anova)

Summary and Conclusion

This tutorial demonstrates how to perform ANOVA in Python using manual calculations and packages like statsmodels and pingouin. ANOVA is a powerful tool for comparing means across multiple groups, applicable in fields like psychology, biology, and economics. Always verify assumptions (normality, homogeneity of variances, independence) and use post-hoc tests to identify specific group differences. Effect sizes should also be considered for practical significance.

Resources

  1. ANOVA in Python using Statsmodels
  2. Pingouin: Statistics in Python
  3. ANOVA in Python: A Step-by-Step Guide