Automated Exploratory Data Analysis (EDA) in Python

This tutorial introduces automated Exploratory Data Analysis (EDA) using Python libraries designed to streamline data exploration and visualization. Automated EDA tools generate comprehensive reports and visualizations with minimal code, allowing users to quickly understand dataset characteristics, including distributions, correlations, and missing values. This guide covers ydata-profiling, sweetviz, dtale, and autoviz, demonstrating how to use these libraries to perform efficient EDA. The tutorial assumes basic familiarity with Python and pandas.

Prerequisites

Install Required Packages

The following libraries will be used:

  • ydata-profiling: Generates interactive HTML EDA reports with statistics, correlations, and missing value analysis.
  • sweetviz: Creates comparison reports for datasets or feature-target distributions.
  • dtale: Provides an interactive web interface for exploring DataFrames.
  • autoviz: Automatically generates visualizations for all variables in a dataset.

Install the packages using pip:

pip install pandas ydata-profiling sweetviz dtale autoviz

Verify Installation

Check if the packages are installed:

Code 
from importlib import metadata

packages = ["pandas", "ydata-profiling", "sweetviz", "dtale", "autoviz"]
installed = {}
for pkg in packages:
    try:
        installed[pkg] = metadata.version(pkg)
    except metadata.PackageNotFoundError:
        pass
print("Installed packages:", installed)

Import Libraries

Import the required libraries with suppressed warnings for cleaner output:

Code 
import os
import warnings
warnings.filterwarnings("ignore")

# Avoid dtale grabbing a commonly busy port range.
os.environ.setdefault("DTALE_MIN_PORT", "49100")
os.environ.setdefault("DTALE_MAX_PORT", "49999")

import pandas as pd
from ydata_profiling import ProfileReport
import sweetviz as sv
import dtale
from autoviz.AutoViz_Class import AutoViz_Class

Dataset

We will use a sample dataset, gp_soil_data_na.csv, from a GitHub repository for demonstration. This dataset contains soil-related variables, including SOC (Soil Organic Carbon), DEM, and NLCD (land cover categories).

Code 
# Load dataset
url = "https://github.com/zia207/Python_for_Beginners/raw/refs/heads/main/Data/gp_soil_data_na.csv"
df = pd.read_csv(url)

ydata-profiling: Comprehensive EDA Report

ydata-profiling generates an interactive HTML report summarizing dataset characteristics, including variable types, quantiles, correlations, and missing values.

Generate and Save Report

Code 
# Disable word-level summaries to avoid wordcloud/TrueType font issues in headless environments.
profile = ProfileReport(
    df,
    title="Soil Data EDA Report",
    explorative=True,
    vars={"text": {"words": False}, "cat": {"words": False}},
)

# Save to HTML file
profile.to_file("soil_data_profile_report.html")

This code produces an HTML file (soil_data_profile_report.html) with sections for: - Dataset overview (rows, columns, missing values) - Variable summaries (distributions, statistics) - Correlation matrices (Pearson, Spearman, etc.) - Missing value patterns - Sample data preview

Open the HTML file in a browser to explore the interactive report.

Key Features

  • Interactive: Clickable tabs for variable details and correlations.
  • Comprehensive: Includes histograms, boxplots, and missing value visualizations.
  • Customizable: Supports minimal mode for large datasets or custom correlation methods.

sweetviz: Comparative EDA Reports

sweetviz generates reports focused on dataset comparisons or feature-target analysis, producing visually appealing HTML outputs.

Generate Report for Single Dataset

Code 
import sweetviz as sv

# Analyze the dataset with error handling
try:
    report = sv.analyze(df)
    # Save to HTML file
    report.show_html("soil_data_sweetviz_report.html")
except AttributeError as e:
    print("Error: Likely a compatibility issue with NumPy or Sweetviz. Try updating Sweetviz (`pip install --upgrade sweetviz`) or downgrading NumPy (`pip install numpy==1.23.5`).")
    print(f"Error details: {e}")

dtale: Interactive Web Interface

dtale provides a web-based interface for exploring DataFrames, allowing filtering, sorting, and chart creation interactively.

Launch dtale Interface

Code 
# Keep rendering non-interactive; then stop dtale so batch runs can exit.
d = dtale.show(df, open_browser=False)
if d is not None:
    print(d.main_url())
    d.kill()

This opens a web browser with an interactive interface where you can: - View and filter the DataFrame - Generate plots (histograms, scatter plots, boxplots) - Calculate summary statistics - Analyze correlations and missing values

Key Features

  • Interactive: Real-time filtering and sorting.
  • Charting: Create and customize plots within the interface.
  • Exportable: Export data or charts for further analysis.

To stop the dtale server, close the browser tab or terminate the Python process.

autoviz: Automatic Visualizations

autoviz generates visualizations for all variables in a dataset with minimal configuration, producing charts like histograms, scatter plots, and boxplots.

Generate Visualizations

Code 
# Initialize AutoViz
AV = AutoViz_Class()

# Generate visualizations
report = AV.AutoViz(
    filename="",
    dfte=df,
    depVar="",  # No target variable for unsupervised EDA
    chart_format="html",
    verbose=2  # Save plots to HTML
)
Code 
print(report)

This generates an HTML directory with visualizations for each variable, including: - Histograms for numerical variables - Bar plots for categorical variables - Scatter plots for relationships between variables

Key Features

  • Automated: Minimal code for comprehensive visualizations.
  • Customizable: Supports target variable analysis and chart type selection.
  • Scalable: Handles large datasets with sampling options.

Handling Missing Values

Automated EDA tools also highlight missing values. For example, ydata-profiling and sweetviz include missing value visualizations. To programmatically handle missing values before EDA:

Code 
# Remove rows with missing values
df_clean = df.dropna()

# Impute missing values with median
df_imputed = df.fillna(df.median(numeric_only=True))

# Re-run EDA on cleaned data (e.g., with ydata-profiling)
profile_clean = ProfileReport(
    df_clean,
    title="Cleaned Soil Data EDA Report",
    vars={"text": {"words": False}, "cat": {"words": False}},
)
profile_clean.to_file("soil_data_cleaned_profile_report.html")

Summary and Conclusion

Automated EDA tools like ydata-profiling, sweetviz, dtale, and autoviz simplify the process of exploring and visualizing datasets. These libraries provide quick insights into data distributions, correlations, and missing values, saving time compared to manual EDA. By generating interactive reports and visualizations, they enable both beginners and experienced analysts to uncover patterns and anomalies efficiently. Experiment with these tools on diverse datasets to leverage their strengths for specific use cases, such as comparing subgroups or exploring large datasets interactively.

References

  1. ydata-profiling Documentation
  2. sweetviz Documentation
  3. dtale Documentation
  4. autoviz Documentation