Basic Data Exploration and Visualization in Python

This tutorial provides a comprehensive guide to exploratory data analysis (EDA) and visualization using Python. It covers essential concepts, techniques, and tools for importing, cleaning, preprocessing, and visualizing data to gain meaningful insights. You will learn how to manipulate data, perform statistical analyses, and interpret results effectively. Whether you’re a beginner or an experienced data analyst, this tutorial equips you with the skills to explore and visualize data using Python.

Check and Install Required Packages

The following Python libraries will be used in this tutorial:

  • pandas: For data manipulation and analysis.
  • numpy: For numerical computations.
  • matplotlib: For creating static, animated, and interactive visualizations.
  • seaborn: For statistical data visualization based on matplotlib.
  • scipy: For scientific and statistical computations.
  • statsmodels: For statistical modeling and testing.
  • pingouin: For additional statistical tests.
  • tabulate: For creating publication-ready tables.
  • missingno: For visualizing missing data.
  • plotly: For interactive visualizations.

To install these packages, run the following command in your Python environment:

pip install pandas numpy matplotlib seaborn scipy statsmodels pingouin tabulate missingno plotly

Verify Installation

To ensure all required packages are installed, you can check their availability:

Code 
import pkg_resources

packages = ['pandas', 'numpy', 'matplotlib', 'seaborn', 'scipy', 'statsmodels', 'pingouin', 'tabulate', 'missingno', 'plotly']
installed = {pkg: pkg_resources.get_distribution(pkg).version for pkg in packages if pkg in [p.key for p in pkg_resources.working_set]}
print("Installed packages:", installed)

Import Packages

Import the required libraries with suppressed warnings for a cleaner output:

Code 
import warnings
warnings.filterwarnings('ignore')

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from scipy import stats
import statsmodels.api as sm
import pingouin as pg
from tabulate import tabulate
import missingno as msno
import plotly.express as px

Data

We will use the same datasets as in the R tutorial, available from GitHub:

Code 
# Import datasets
mf_na = pd.read_csv("https://github.com/zia207/Python_for_Beginners/raw/refs/heads/main/Data/gp_soil_data_na.csv")
as_df = pd.read_csv("https://github.com/zia207/Python_for_Beginners/raw/refs/heads/main/Data/rice_arsenic_data.csv")

Data Structure

The info() method in pandas provides a concise summary of the DataFrame, including column names, data types, and missing values.

Code 
# Display data structure
mf_na.info()

Missing Values

Missing values can affect the accuracy of data analysis. The isna() method identifies missing values in a DataFrame.

Code 
# Total number of missing values
print("Total missing values:", mf_na.isna().sum().sum())

# Column names with missing values
list_na = mf_na.columns[mf_na.isna().any()].tolist()
print("Columns with missing values:", list_na)

Visualizing Missing Values

The missingno library can visualize missing data patterns:

Code 
msno.matrix(mf_na)
plt.title("Missing Data Matrix")
plt.show()

Handling Missing Values

Common strategies for handling missing values include deletion and imputation. Here, we demonstrate both:

Remove Missing Values

Use dropna() to remove rows with missing values:

Code 
mf = mf_na.dropna()
print("Total missing values after dropping:", mf.isna().sum().sum())

Impute Missing Values

Impute missing values with the median of the column:

Code 
# Identify columns with missing values
list_na = mf_na.columns[mf_na.isna().any()].tolist()
# Impute with median
median_missing = mf_na[list_na].median()
mf_imp = mf_na.copy()
for col in list_na:
    mf_imp[col] = mf_imp[col].fillna(median_missing[col])
print(mf_imp.head())

Visual Inspection of Data Distribution

Visualizing data distribution is crucial for understanding patterns, central tendencies, and spread. Common techniques include histograms, kernel density plots, and quantile-quantile (Q-Q) plots.

Histograms

Histograms display the frequency distribution of numerical data, helping identify patterns, outliers, and skewness.

Code 
### Using Matplotlib
plt.hist(mf['SOC'], bins=30)
plt.title("Histogram of Soil SOC")
plt.xlabel("Soil SOC (mg/g)")
plt.ylabel("Frequency")
plt.show()
Code 
### Using Seaborn
sns.histplot(data=mf, x='SOC')
plt.title("Histogram of Soil SOC")
plt.xlabel("Soil SOC (mg/g)")
plt.ylabel("Frequency")
plt.show()

Kernel Density Plots

Kernel density plots provide a smoothed estimate of the data’s probability density function.

Code 
### Using Matplotlib/Seaborn
sns.kdeplot(data=mf, x='SOC')
plt.title("Kernel Density of Soil SOC")
plt.xlabel("Soil SOC (mg/g)")
plt.ylabel("Density")
plt.show()

Q-Q Plot

Q-Q plots compare the quantiles of a dataset against a theoretical distribution (e.g., normal distribution).

Code 
stats.probplot(mf['SOC'], dist="norm", plot=plt)
plt.title("Q-Q Plot of Soil SOC from a Normal Distribution")
plt.xlabel("Theoretical Quantiles")
plt.ylabel("Sample Quantiles")
plt.show()

Normality Tests

Normality tests assess whether data follows a normal distribution. Common tests include Shapiro-Wilk, Anderson-Darling, and Kolmogorov-Smirnov.

Shapiro-Wilk Test

Code 
stat, p_value = stats.shapiro(mf['SOC'])
print("Shapiro-Wilk Test:")
print(f"Statistic: {stat:.4f}, p-value: {p_value:.4f}")

# A p-value < 0.05 suggests the data is not normally distributed.

Anderson-Darling Test

Code 
result = stats.anderson(mf['SOC'], dist='norm')
print("Anderson-Darling Test:")
print(f"Statistic: {result.statistic:.4f}")
for i, (crit_val, sig_level) in enumerate(zip(result.critical_values, result.significance_level)):
    print(f"Critical value at {sig_level}%: {crit_val:.4f}")

Kolmogorov-Smirnov Test

Code 
stat, p_value = stats.kstest(mf['SOC'], 'norm', args=(mf['SOC'].mean(), mf['SOC'].std()))
print("Kolmogorov-Smirnov Test:")
print(f"Statistic: {stat:.4f}, p-value: {p_value:.4f}")

Skewness and Kurtosis

Skewness measures the asymmetry of a distribution, while kurtosis measures its peakedness.

Code 
from scipy.stats import skew, kurtosis

print("Skewness:", skew(mf['SOC'], nan_policy='omit'))
print("Kurtosis:", kurtosis(mf['SOC'], nan_policy='omit'))

# A high positive skewness indicates a right-skewed distribution, and a high positive kurtosis suggests a peaked distribution.

Data Transformation for Normality

Data transformations can help achieve normality by adjusting the data’s scale or shape.

Logarithmic Transformation

Code 
mf['log_SOC'] = np.log10(mf['SOC'])
plt.hist(mf['log_SOC'], bins=30)
plt.title("Histogram of Log Soil SOC")
plt.xlabel("Log Soil SOC")
plt.ylabel("Frequency")
plt.show()
print("Shapiro-Wilk Test for Log SOC:", stats.shapiro(mf['log_SOC']))

Square Root Transformation

Code 
mf['sqrt_SOC'] = np.sqrt(mf['SOC'])
plt.hist(mf['sqrt_SOC'], bins=30)
plt.title("Histogram of Square Root Soil SOC")
plt.xlabel("Sqrt Soil SOC")
plt.ylabel("Frequency")
plt.show()
print("Shapiro-Wilk Test for Sqrt SOC:", stats.shapiro(mf['sqrt_SOC']))

Inverse Transformation

Code 
mf['inv_SOC'] = 1 / mf['SOC']
plt.hist(mf['inv_SOC'], bins=30)
plt.title("Histogram of Inverse Soil SOC")
plt.xlabel("Inverse Soil SOC")
plt.ylabel("Frequency")
plt.show()
print("Shapiro-Wilk Test for Inverse SOC:", stats.shapiro(mf['inv_SOC']))

Box-Cox Transformation

Code 
from scipy.stats import boxcox

mf['bc_SOC'], lambda_param = boxcox(mf['SOC'])
plt.hist(mf['bc_SOC'], bins=30)
plt.title("Histogram of Box-Cox Soil SOC")
plt.xlabel("Box-Cox Soil SOC")
plt.ylabel("Frequency")
plt.show()
print("Box-Cox Lambda:", lambda_param)
print("Shapiro-Wilk Test for Box-Cox SOC:", stats.shapiro(mf['bc_SOC']))

Outliers

Outliers are data points significantly different from others, potentially skewing analysis results.

Visual Inspection with Boxplot

Code 
### Using Matplotlib
plt.boxplot(mf['SOC'])
plt.title("Boxplot of Soil SOC")
plt.ylabel("Soil SOC (mg/g)")
plt.show()
Code 
### Using Seaborn
sns.boxplot(x=mf['SOC'])
plt.title("Boxplot of Soil SOC")
plt.xlabel("Soil SOC (mg/g)")
plt.show()

Count Outliers

Code 
Q1 = mf['SOC'].quantile(0.25)
Q3 = mf['SOC'].quantile(0.75)
IQR = Q3 - Q1
outliers = mf['SOC'][(mf['SOC'] < (Q1 - 1.5 * IQR)) | (mf['SOC'] > (Q3 + 1.5 * IQR))]
print("Number of outliers:", len(outliers))

Box-Jitter Plot

Code 
plt.figure(figsize=(8, 6))
sns.boxplot(y='NLCD', x='SOC', data=mf, color='white')
sns.stripplot(y='NLCD', x='SOC', hue='MAT', size=4, palette='viridis', data=mf, jitter=True)
plt.title("Variability of Soil SOC in Relation to MAT in Four NLCD")
plt.xlabel("Soil SOC (mg/g)")
plt.ylabel("")
plt.legend(title="MAT")
plt.show()

Tukey’s Method

Code 
def detect_outliers(x):
    if not np.issubdtype(x.dtype, np.number):
        raise ValueError("Data provided must be numeric")
    Q1 = np.quantile(x, 0.25)
    Q3 = np.quantile(x, 0.75)
    IQR = Q3 - Q1
    lower_limit = Q1 - 1.5 * IQR
    upper_limit = Q3 + 1.5 * IQR
    lower_outliers = x[x < lower_limit]
    upper_outliers = x[x > upper_limit]
    print(f"Lower Limit: {lower_limit}")
    print(f"Upper Limit: {upper_limit}")
    print(f"Lower range outliers: {lower_outliers.values}")
    print(f"Upper range outliers: {upper_outliers.values}")

detect_outliers(mf['SOC'])

Z-Score Method

Code 
from scipy.stats import zscore

z_scores = zscore(mf['SOC'])
outliers = mf['SOC'][np.abs(z_scores) > 3]
print("Outliers using Z-score method:", outliers.values)

Descriptive Statistics

Descriptive statistics summarize dataset characteristics, including measures of central tendency and dispersion.

Code 
# Basic summary statistics
print(mf.describe())

Summary Table with Pandas

Code 
summary_stat = mf[['SOC', 'DEM', 'Slope', 'Aspect', 'TPI', 'KFactor', 'MAP', 'MAT', 'NDVI', 'SiltClay']].agg(
    ['min', 'quantile', 'median', 'max', 'mean', 'std']
).round(2).T.reset_index()
summary_stat.columns = ['Variable', 'Min', 'Quantile ', 'Median',  'Max', 'Mean', 'SD']
print(tabulate(summary_stat, headers='keys', tablefmt='pretty'))

Summary Statistics by Group

Code 
summarise_soc = mf.groupby('NLCD')['SOC'].agg(
    Mean='mean',
    Median='median',
    Min='min',
    Max='max',
    SD='std',
    SE=lambda x: x.std() / np.sqrt(len(x))
).round(3).reset_index()
print(tabulate(summarise_soc, headers='keys', tablefmt='pretty'))

Barplot

Code 
plt.figure(figsize=(8, 6))
sns.barplot(x='Mean', y='NLCD', data=summarise_soc, color='gray')
plt.errorbar(y=summarise_soc.index, x=summarise_soc['Mean'], xerr=summarise_soc['SE'], fmt='none', c='black')
plt.title("Mean ± SE of Soil SOC")
plt.xlabel("Soil SOC (mg/g)")
plt.ylabel("")
plt.show()

T-Test

One-Sample T-Test

Code 
np.random.seed(42)
sample_data = as_df['GAs'].sample(50)
sample_mean = sample_data.mean()
population_mean = as_df['GAs'].mean()
t_stat, p_value = stats.ttest_1samp(sample_data, population_mean)
print("One-Sample T-Test:")
print(f"Sample Mean: {sample_mean:.3f}")
print(f"Population Mean: {population_mean:.3f}")
print(f"T-statistic: {t_stat:.3f}, p-value: {p_value:.3f}")

Two-Sample T-Test

Code 
low = as_df[as_df['TREAT_ID'] == 1]['GY']
high = as_df[as_df['TREAT_ID'] == 2]['GY']
t_stat, p_value = stats.ttest_rel(low, high, alternative='greater')
print("Two-Sample T-Test:")
print(f"T-statistic: {t_stat:.3f}, p-value: {p_value:.3f}")

Summary and Conclusion

This tutorial provides a comprehensive guide to exploratory data analysis and visualization in Python, leveraging powerful libraries like pandas, matplotlib, seaborn, and scipy. It covers data import, cleaning, visualization, and statistical analysis, enabling users to uncover patterns, trends, and outliers in their datasets. Visualization techniques, including histograms, density plots, Q-Q plots, and boxplots, are demonstrated to enhance data understanding. Statistical tests and transformations ensure robust analysis, while summary tables and plots facilitate clear communication of findings. Practitioners are encouraged to experiment with diverse datasets and visualization techniques to deepen their data exploration skills.

References