Simple Linear Regression Analysis

Simple Linear Regression is widely used in various fields for prediction, understanding relationships between variables, and making informed decisions based on observed data. In this tutorial, we will guide you through performing simple linear regression analysis step by step. We’ll cover everything you need to know, from understanding the underlying concepts to fitting regression models and interpreting the results using Python, a versatile programming language for statistical analysis and data visualization.

This tutorial is designed for beginners and assumes no prior knowledge of regression analysis or Python programming. By the end of this tutorial, you will have a solid understanding of simple linear regression and be able to apply it to your own datasets. We will provide clear explanations, practical examples, and hands-on exercises to help you grasp the concepts effectively.

Introduction

Regression analysis is a powerful statistical tool that allows us to explore the complex relationships between variables. It helps us to understand how changes in one variable affect the other, quantify the strength of the relationship, and make predictions about future outcomes. By examining the data, we can identify patterns, trends, and anomalies that may not appear at first glance. This method is commonly used in economics, psychology, and finance to analyze large datasets and draw meaningful insights. With its ability to model complex relationships, regression analysis is an essential tool for any data scientist or researcher.

Linear regression is a statistical methodology that enables modeling the relationship between a dependent variable and one or more independent variables. The aim is to identify the linear relationship that best predicts the dependent variable based on the values of the independent variables.

Linear regression models are widely used in various fields, including economics, finance, biology, and social sciences, to analyze the relationship between variables and predict future outcomes. The technique is instrumental in identifying trends, patterns, and associations between variables, which can be used to develop hypotheses and test theories.

The ultimate objective of linear regression is to identify the linear equation that best describes the relationship between the independent and dependent variables. The equation can then be used to predict the dependent variable’s value based on the independent variables’ values. The model’s accuracy can be measured using various statistical measures, such as the coefficient of determination (R-squared) and the root mean squared error (RMSE). These measures indicate how well the model fits the data and how accurate the predictions are.

In conclusion, linear regression is a powerful statistical tool that can be used to gain insights into the relationship between variables, make predictions, and test hypotheses. Its wide range of applications and the accuracy of its predictions make it an indispensable tool in various fields.

Here are the key components of linear regression:

  1. Dependent Variable (Y): This is the variable we want to predict or explain. It’s also known as the response variable.
  2. Independent Variable(s) (X): These are the variables used to predict the dependent variable. If there’s only one independent variable, it’s a simple linear regression. If there are multiple, it’s a multiple linear regression.
  3. Regression Equation: The linear regression equation is represented as \[ Y = b_0 +b_1X_1 + b_2X_2 + \ldots +b_n.X_n + \varepsilon \] where \(Y\) is the dependent variable, \(X_1, X_2,...X_n\) are the independent variables, \(b_0\) is the intercept, \(b_1, b_2,...b_n\) are coefficient and \(\varepsilon\) is the error term.
  4. Coefficients: These are the weights assigned to each independent variable, representing the strength and direction of their impact on the dependent variable.
  5. Residuals: Residuals are the differences between the actual and predicted values. The goal is to minimize these differences.
  6. Assumptions: Linear regression assumes linearity, independence, homoscedasticity, and normality of residuals.
  7. Ordinary Least Squares (OLS): This method is used to find the coefficients by minimizing the sum of squared residuals.

Linear regression analysis is based on six fundamental assumptions:

  • The dependent and independent variables show a linear relationship between the slope and the intercept.
  • The independent variable is not random.
  • The value of the residual (error) is zero.
  • The value of the residual (error) is constant across all observations.
  • The value of the residual (error) is not correlated across all observations.
  • The residual (error) values follow the normal distribution

Linear regression is widely used in various fields, such as economics, finance, biology, etc. It’s a simple yet powerful tool for predicting and understanding relationships between variables.

Simple Linear Regression

Simple Linear Regression is a statistical method used to model the relationship between a single dependent variable (usually denoted as \(Y\)) and a single independent variable (usually denoted as \(X\)). The relationship is assumed to be linear and is represented by the equation of a straight line.

The general form of the simple linear regression model is given by: \[ Y = b_0 +b_1X + \varepsilon \] where \(Y\) is the dependent variable, \(X\) is the independent variable, \(b_0\) is the intercept, \(b_1\) is slope and \(\varepsilon\) is the error term.

The goal of simple linear regression is to estimate the values of \(b_0\) and \(b_1\) that minimize the sum of squared differences between the observed values of \(Y\) and the values predicted by the model.

The estimated coefficients, denoted as (\(\hat{b_0}\)) and (\(\hat{b_1}\)), are typically computed using the least squares method. Once the coefficients are estimated, the regression equation becomes: \[ \hat{Y} = \hat{b_0}+\hat{b_1}X \] This equation can be used to make predictions for the dependent variable \(Y\) based on new values of \(X\).

The goodness of fit of the model is often assessed using metrics such as the coefficient of determination (\(R^2\)), which indicates the proportion of the variance in the dependent variable that is explained by the independent variable. Here below formula for \(R^2\) is given by: \[ R^2 = 1 - \frac{SS_{res}}{SS_{tot}} \] where \(SS_{res}\) is the sum of squared residuals and \(SS_{tot}\) is the total sum of squares.

Simple Regression Analysis from Scratch

In this section, we will perform a simple linear regression analysis from scratch using Python. We will create a dataset, calculate the coefficients for the regression line, make predictions, and evaluate the model’s performance.

Code 
import numpy as np
import pandas as pd

# Create a simple dataset
x = np.array([1, 2, 3, 4, 5])
y = np.array([2, 4, 5, 4, 5])

# Calculate the means of x and y
mean_x = np.mean(x)
mean_y = np.mean(y)

# Calculate the coefficients for the regression line: y = m * x + b
numerator = np.sum((x - mean_x) * (y - mean_y))
denominator = np.sum((x - mean_x)**2)
m = numerator / denominator
b = mean_y - m * mean_x

# Display the coefficients
print("Slope (m):", m)
print("Intercept (b):", b)

# Make predictions using the regression line
y_pred = m * x + b

# Calculate R-squared (R²)
ss_total = np.sum((y - mean_y)**2)  # Total sum of squares
ss_residual = np.sum((y - y_pred)**2)  # Residual sum of squares
r_squared = 1 - (ss_residual / ss_total)

# Display R² value
print("R-squared (R²):", r_squared)

# Display summary statistics
summary_stats = pd.DataFrame({
    'x': x,
    'y': y,
    'y_pred': y_pred,
    'residuals': y - y_pred
})
print(summary_stats)

Simple Regression Analysis with Python

Simple linear regression analysis is a powerful statistical technique used to model the relationship between a dependent variable and one or more independent variables. In this tutorial, we will explore how to perform simple linear regression analysis using Python, a popular programming language for statistical computing and data analysis. We will cover the following topics:

Check and Install Required Python Packages

In this exercise, we will use the following Python packages:

  • pandas: For data manipulation and analysis.
  • numpy: For numerical computations.
  • statsmodels: For estimating statistical models, performing tests, and statistical visualizations.
  • scikit-learn: For machine learning and additional metrics.
  • matplotlib: For creating static, animated, and interactive visualizations.
  • seaborn: For statistical data visualization built on matplotlib.
Code 
import importlib.util
import sys

# List of required packages
packages = ['numpy', 'pandas', 'statsmodels', 'scikit-learn', 'matplotlib', 'seaborn']

# Check and install missing packages
# Check and install missing packages
for package in packages:
    if not importlib.util.find_spec(package):
        try:
            import pip
            pip.main(['install', package])
        except ImportError:
            print(f"Failed to install {package}. Pip is not available.")

Verify Installation

Code 
# Verify package availability
for package in packages:
    print(f"{package} installed: {bool(importlib.util.find_spec(package))}")

Load Packages

Code 
import numpy as np
import pandas as pd
import statsmodels.api as sm
from sklearn.metrics import mean_squared_error, r2_score
import matplotlib.pyplot as plt
import seaborn as sns

Check Loaded Packages

Code 
# Check loaded packages (simple print of imports)
print("Successfully loaded packages: numpy, pandas, statsmodels, scikit-learn, matplotlib, seaborn")

Data

We will use the read_csv() function from the pandas package to import data as a tidy DataFrame.

Code 
# Load data
mf = pd.read_csv("https://github.com/zia207/Python_for_Beginners/raw/refs/heads/main/Data/gp_soil_data.csv")
print(mf.head())

Fit a Simple Linear Model

In this exercise, we will fit a simple linear regression model to explore the relationship between soil organic carbon (SOC) and Normalized Vegetation Index (NDVI) using the OLS function from statsmodels:

Code 
# Fit the regression model: SOC ~ NDVI
X = sm.add_constant(mf['NDVI'])  # Add intercept term
slm_soc = sm.OLS(mf['SOC'], X).fit()

# Display the summary
print(slm_soc.summary())

The summary() method displays the summary of the linear model, including the estimated coefficients, standard errors, t-statistics, p-values, and \(R^2\) value.

Create Regression Summary Table

The summary() from statsmodels provides a clean regression output table. For a more tidy DataFrame-like output, you can extract parameters:

Code 
# Tidy summary table
tidy_summary = pd.DataFrame({
    'term': slm_soc.params.index,
    'estimate': slm_soc.params.values,
    'std_error': slm_soc.bse.values,
    't_value': slm_soc.tvalues,
    'p_value': slm_soc.pvalues
})
print(tidy_summary)

Model Performance

The statsmodels package provides utilities for computing indices of model quality and goodness of fit. These include measures like R-squared, AIC, BIC, RMSE, but also functions to check models for assumptions like normality or heteroscedasticity.

Model Performance Summaries

slm_soc.summary() computes indices of model performance for regression models. Typical indices include R-squared, AIC, BIC, and RMSE.

Code 
# Model performance from summary (or extract)
print(f"AIC: {slm_soc.aic}")
print(f"BIC: {slm_soc.bic}")
print(f"R-squared: {slm_soc.rsquared}")
print(f"Adjusted R-squared: {slm_soc.rsquared_adj}")
print(f"RMSE: {np.sqrt(slm_soc.mse_resid)}")
print(f"Sigma (Residual Standard Error): {slm_soc.scale**0.5}")
Note

AIC: AIC (Akaike Information Criterion) is a statistical measure used for model selection. It is a way to balance the goodness of fit of a model with the number of parameters in the model.

AICc: AICc (corrected Akaike Information Criterion) is a modification of the AIC statistic that is used to adjust for small sample sizes. AICc is calculated using the formula:

BIC: BIC (Bayesian Information Criterion) is another statistical measure used for model selection, similar to AIC. BIC is also based on the trade-off between model fit and model complexity, but it uses a different penalty for the number of parameters in the model.

RMSE: Root Mean Square Error (RMSE) is a commonly used metric to measure the accuracy of a predictive model. It is used to quantify the difference between the predicted values and the actual values in a dataset. RMSE is calculated by taking the square root of the mean of the squared differences between the predicted and actual values.

\(R^2\): \(R^2\), is a statistical measure that indicates how well a regression model fits the data. It is also known as the coefficient of determination.

Adjusted \(R^2\): Adjusted R2 is a modified version of the R2 statistic that adjusts for the number of independent variables in the regression model. It is used to evaluate the goodness of fit of a regression model, while accounting for the complexity of the model.

Sigma: The sigma of a regression model, also known as the residual standard error, is a measure of the variability of the errors in the regression model. It is a standard deviation of the residuals, which are the differences between the predicted values and the actual values.

R-squared and RMSE

scikit-learn has generic functions like r2_score and mean_squared_error to compute \(R^2\) and RMSE for models.

Code 
# Predictions
y_pred = slm_soc.predict(X)

# R2
r2 = r2_score(mf['SOC'], y_pred)
print("R2:", r2)

# RMSE
rmse = np.sqrt(mean_squared_error(mf['SOC'], y_pred))
print("RMSE:", rmse)

Visualization of Model Assumptions

The statsmodels package provides functions to check model assumptions, like plot_diagnostics(). To get a comprehensive check, use it directly.

Code 
# Comprehensive assumption checks
fig = sm.graphics.plot_regress_exog(slm_soc, 'NDVI')
plt.show()
Note

Posterior predictive: Posterior predictive checks for a linear model involve simulating data sets from the posterior distribution of the model parameters, and then comparing the simulated data sets to the observed data

Linearity: Linearity plots the residuals (i.e., the differences between the observed values and the predicted values) against the independent variable(s). If the plot shows a random pattern with no clear trend, then the linearity assumption is likely to hold. If the plot shows a pattern or trend, then the linearity assumption may be violated.

Homogeneity of variance: Homogeneity of variance can be assessed by examining a scatter plot of the residuals against the predicted values or against the independent variable(s). If the plot shows a random pattern with no clear trend, then the assumption of homogeneity of variance is likely to hold. If the plot shows a pattern or trend, such as increasing or decreasing variance, then the assumption of homogeneity of variance may be violated.

Influential observation : The influential observations in a linear regression model is an observation that has a strong effect on the estimated regression coefficients. These observations can have a large impact on the regression model and can affect the conclusions that are drawn from the analysis. Cook’s distance or the leverage statistic is generally use to measure statistical measures of influence. Observations that have high values of Cook’s distance or leverage may be influential.

Normality of residuals: It can be assessed by examining a histogram or a normal probability plot of the residuals. If the histogram shows a symmetric bell-shaped distribution, and the normal probability plot shows a roughly straight line, then the assumption of normality is likely to hold. If the histogram or normal probability plot shows departures from normality, such as skewness or outliers, then the assumption of normality may be violated.

Visualize Linear Model

seaborn and matplotlib extend visualization capabilities for scatter plots of two variables. We can add a regression line using sns.regplot and annotate with the equation and R-squared.

Code 
# Scatter plot with regression line and annotations
plt.figure(figsize=(4.5, 4))
sns.scatterplot(data=mf, x='NDVI', y='SOC', s=20)

# Add regression line
sns.regplot(data=mf, x='NDVI', y='SOC', scatter=False, color='red')

# Annotate with equation and adjusted R2
intercept = slm_soc.params[0]
slope = slm_soc.params[1]
adj_r2 = slm_soc.rsquared_adj

plt.text(0.05, 0.95, f'y = {slope:.2f}x + {intercept:.2f}\nAdj. R² = {adj_r2:.3f}', 
         transform=plt.gca().transAxes, fontsize=10, verticalalignment='top',
         bbox=dict(boxstyle='round', facecolor='wheat', alpha=0.5))

plt.title("Scatterplot of Soil SOC vs NDVI")
plt.xlabel("NDVI")
plt.ylabel("Soil OC (mg/g)")
plt.tight_layout()
plt.show()

Summary and Conclusion

This tutorial on simple linear regression analysis in Python offers a comprehensive guide to help you understand and apply this statistical technique. By following this tutorial, you will gain a detailed understanding of the process involved in simple linear regression analysis. You will learn how to confidently handle data using Python and perform regression analysis to identify relationships between variables. This tutorial will equip you with the knowledge and skills to make accurate predictions and draw valuable insights from your data.

Simple linear regression analysis is an essential tool in your data analysis toolkit. It enables you to investigate trends and patterns in your data and effectively inform decision-making processes. You will learn to interpret regression output, assess model fit and performance, and make predictions using your model. Additionally, you will understand how to test for assumptions, identify influential observations, and visualize them.

By the end of this tutorial, you will thoroughly understand how to use simple linear regression analysis in Python. You will be well-equipped to explore relationships between variables, make predictions, and draw valuable insights from your data. Simple linear regression analysis is a versatile statistical technique that can be used across various industries, making it an indispensable tool for data analysts and researchers alike.

Resources

  1. Regression Basics in Python
  2. Statsmodels Documentation
  3. Seaborn Regression Plots
  4. Scikit-learn Metrics