Descriptive Statistics

This tutorial provides a comprehensive introduction to descriptive statistics using the Python programming language. It covers essential concepts such as mean, median, mode, range, variance, standard deviation, quantiles, and interquartile range (IQR). By the end of this tutorial, you will be able to compute and interpret these fundamental statistical measures using real-world data in Python — ideal for beginners and those refreshing their skills.

💡 Note: While R is widely used in statistics, Python offers powerful, flexible tools for data analysis. This guide bridges the gap by translating R-based descriptive statistics into Python equivalents.

Introduction

Descriptive Statistics is the branch of statistics focused on summarizing and describing the main features of a dataset. It helps answer questions like: - What is the typical value? (Central Tendency) - How spread out are the values? (Dispersion) - Are there extreme or unusual values? (Outliers)

Key measures include: - Mean, Median, Mode → Central tendency - Range, Variance, Standard Deviation, IQR → Dispersion - Quantiles → Distribution shape

These metrics enable data analysts to make sense of raw numbers, detect patterns, and communicate insights effectively — without advanced modeling.

We’ll use the rice arsenic dataset (same as in the R tutorial) to demonstrate all concepts with real data.

Prerequisites

Install the required packages:

Code 
import importlib.util
import sys

# List of required packages
packages = ['pandas', 'scipy']

# Check and install missing packages
for package in packages:
    if not importlib.util.find_spec(package):
        try:
            import pip
            pip.main(['install', package])
        except ImportError:
            print(f"Failed to install {package}. Pip is not available.")

# Import packages
import pandas as pd
import numpy as np
from scipy import stats
Code 
# Verify package availability
for package in packages:
    print(f"{package} installed: {bool(importlib.util.find_spec(package))}")

Data

Code 
# Load the dataset directly from GitHub
url = "https://github.com/zia207/Python_for_Beginners/raw/refs/heads/main/Data/rice_arsenic_data.csv"
df = pd.read_csv(url)

# Display first few rows to inspect
print(df.head())

Central Tendency

Central tendency is a statistical measure that assists in describing the center point of a set of data values. This concept is used to identify a single value that is considered most representative of the entire distribution. By determining the central tendency, we can gain insights into the typical or common values in a dataset. Mean, median, and mode are the three most commonly used measures of central tendency.

Mean

The mean is the arithmetic average: sum of all values divided by count.

\[ \bar{X} = \frac{\sum_{i=1}^{n} X_i}{n} \]

Code 
# Overall mean of Grain Yield (GY)
mean_gy = df['GY'].mean()
print(f"Overall Mean Grain Yield: {mean_gy:.4f}")

# Mean by soil treatment group (TREAT)
mean_by_treat = df.groupby('TREAT')['GY'].mean()
print("\nMean Grain Yield by Soil Treatment:")
print(mean_by_treat)

Median

The median is the middle value when data is sorted. It is robust to outliers.

  • Odd n: Middle value
  • Even n: Average of two middle values
Code 
# Overall median of GY
median_gy = df['GY'].median()
print(f"\nOverall Median Grain Yield: {median_gy:.4f}")

# Median by treatment group
median_by_treat = df.groupby('TREAT')['GY'].median()
print("\nMedian Grain Yield by Soil Treatment:")
print(median_by_treat)

Mode

The mode is the most frequently occurring value. A dataset can have one mode (unimodal), multiple modes (bimodal, multimodal), or none.

Unlike R, Python doesn’t have a built-in mode() function for Series, but we can use scipy.stats.mode() or value_counts().

Code 
# Using scipy.stats.mode (returns mode and count)
mode_result = stats.mode(df['GAs'], keepdims=True)
print(f"\nMode of Grain Arsenic (GAs): {mode_result.mode[0]:.4f} (appears {mode_result.count[0]} times)")

# Alternative: Using pandas value_counts()
mode_pandas = df['GAs'].value_counts().idxmax()
mode_count = df['GAs'].value_counts().max()
print(f"Mode (using value_counts()): {mode_pandas:.4f} (count: {mode_count})")

# Check if multiple modes exist
value_counts = df['GAs'].value_counts()
modes = value_counts[value_counts == value_counts.max()].index.tolist()
if len(modes) > 1:
    print(f"Multiple modes detected: {modes}")
else:
    print("Unimodal dataset.")

Range

The range is the difference between the maximum and minimum values. It gives a quick sense of spread.

\[ \text{Range} = \max(X) - \min(X) \]

Code 
# Calculate range using min and max
range_gy = df['GY'].max() - df['GY'].min()
print(f"\nRange of Grain Yield: {range_gy:.4f}")

# Alternative: using numpy's.ptp() ("peak to peak")
range_gy_np = np.ptp(df['GY'])
print(f"Range (using np.ptp()): {range_gy_np:.4f}")

Variance

Variance measures how far each number in the set is from the mean. Higher variance = more spread.

Sample Variance (most common in practice):

\[ s^2 = \frac{\sum_{i=1}^{n}(X_i - \bar{X})^2}{n-1} \]

Code 
# Sample variance (default in pandas)
var_gy = df['GY'].var()
print(f"\nSample Variance of Grain Yield: {var_gy:.4f}")

# Population variance (divide by n instead of n-1)
pop_var_gy = df['GY'].var(ddof=0)  # ddof=0 for population
print(f"Population Variance: {pop_var_gy:.4f}")

# Manual calculation (for verification)
mean_gy_manual = df['GY'].mean()
squared_diffs = (df['GY'] - mean_gy_manual)**2
manual_var = squared_diffs.sum() / (len(df) - 1)  # sample variance
print(f"Manual sample variance: {manual_var:.4f}")

Standard Deviation

Standard Deviation is the square root of variance. It’s expressed in the same units as the original data, making it more interpretable.

\[ s = \sqrt{s^2} \]

Code 
# Sample standard deviation
std_gy = df['GY'].std()
print(f"\nSample Standard Deviation of Grain Yield: {std_gy:.4f}")

# Population standard deviation
pop_std_gy = df['GY'].std(ddof=0)
print(f"Population Standard Deviation: {pop_std_gy:.4f}")

# Verify: sqrt(variance)
import math
print(f"Verification (sqrt(var)): {math.sqrt(var_gy):.4f}")

Quantiles

Quantiles divide data into equal parts. Common ones include quartiles (Q1, Q2, Q3), deciles, and percentiles.

Code 
# Specific quantiles
q25 = df['GY'].quantile(0.25)
q50 = df['GY'].quantile(0.50)  # same as median
q75 = df['GY'].quantile(0.75)

print(f"\nQuantiles of Grain Yield:")
print(f"25th percentile (Q1): {q25:.4f}")
print(f"50th percentile (Median): {q50:.4f}")
print(f"75th percentile (Q3): {q75:.4f}")

# All quartiles at once
quartiles = df['GY'].quantile([0.25, 0.5, 0.75])
print(f"\nAll Quartiles:\n{quartiles}")

# Deciles (10% intervals)
deciles = df['GY'].quantile(np.arange(0, 1.1, 0.1))
print(f"\nDeciles (0% to 100% in 10% steps):\n{deciles}")

Interquartile Range (IQR)

The IQR is the range between the first (Q1) and third (Q3) quartiles. It’s a robust measure of spread unaffected by outliers.

\[ \text{IQR} = Q3 - Q1 \]

Code 
# Method 1: Subtract quantiles manually
iqr_value = q75 - q25
print(f"\nInterquartile Range (IQR): {iqr_value:.4f}")

# Method 2: Using scipy.stats.iqr (recommended)
from scipy.stats import iqr
iqr_scipy = iqr(df['GY'])
print(f"IQR (using scipy.stats.iqr): {iqr_scipy:.4f}")

# IQR is often used to detect outliers:
lower_bound = q25 - 1.5 * iqr_value
upper_bound = q75 + 1.5 * iqr_value
outliers = df[(df['GY'] < lower_bound) | (df['GY'] > upper_bound)]
print(f"\nNumber of potential outliers (using 1.5*IQR rule): {len(outliers)}")

Summary Statistics

Code 
# Create a summary table for GY column
summary_table = pd.DataFrame({
    'Statistic': ['Mean', 'Median', 'Min', 'Max', 'Range', 'Variance', 'Std Dev', 'Q1', 'Q3', 'IQR'],
    'Value': [
        df['GY'].mean(),
        df['GY'].median(),
        df['GY'].min(),
        df['GY'].max(),
        df['GY'].max() - df['GY'].min(),
        df['GY'].var(),
        df['GY'].std(),
        df['GY'].quantile(0.25),
        df['GY'].quantile(0.75),
        iqr(df['GY'])
    ]
})

print("\n" + "="*50)
print("SUMMARY STATISTICS FOR GRAIN YIELD (GY)")
print("="*50)
print(summary_table.round(4))

Summary and Conclusion

In this tutorial, we’ve covered the core components of descriptive statistics in Python:

Concept Python Function
Mean df['col'].mean()
Median df['col'].median()
Mode stats.mode() or value_counts()
Range max() - min() or np.ptp()
Variance df['col'].var()
Standard Deviation df['col'].std()
Quantiles df['col'].quantile(0.x)
IQR scipy.stats.iqr(df['col'])

Why this matters: Descriptive statistics are the foundation of exploratory data analysis (EDA). They help you: - Spot anomalies - Understand data distribution - Choose appropriate models - Communicate findings clearly

Always visualize alongside numerical summaries (e.g., histograms, boxplots) for full insight.

Resources

Here are excellent resources to deepen your understanding:

  1. Python for Data Analysis by Wes McKinney – Creator of pandas. Chapter 3 covers descriptive stats and data aggregation.
  2. Data Visualization and Descriptive Statistics with Python – Interactive course on DataCamp.
  3. Real Python: Descriptive Statistics with Python – Clear examples with NumPy, SciPy, and pandas.
  4. Kaggle Learn: Data Visualization & Descriptive Stats – Hands-on mini-courses.
  5. Introduction to Statistics with R (adapted for Python) – Though R-focused, the concepts translate directly. Use this guide as your conceptual reference.