Pandas for Beginners

Pandas is a powerful Python library for data manipulation and analysis. This tutorial introduces beginners to Pandas, covering the basics of creating, manipulating, and analyzing data using DataFrames and Series, as well as advanced features like merging, reshaping, time-series analysis, multi-indexing, handling large datasets, and visualization. All examples are simple and include explanations.

Prerequisites

Basic Python knowledge (lists, dictionaries, etc.)
Python and Pandas installed (pip install pandas)
NumPy installed for some advanced features (pip install numpy)
Matplotlib and Seaborn for visualization (pip install matplotlib seaborn)

Importing Pandas

To use Pandas, import it with the common alias pd.

Code

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

Pandas Data Structures

Pandas has two primary data structures: - Series: A one-dimensional array-like object, similar to a list or a column in a table. - DataFrame: A two-dimensional table with rows and columns, like a spreadsheet.

Creating a Series

A Series can be created from a list, dictionary, or array.

Code

# Creating a Series from a list
data = [10, 20, 30, 40]
series = pd.Series(data, index=['a', 'b', 'c', 'd'])
print(series)

Output:

a    10
b    20
c    30
d    40
dtype: int64

The index parameter assigns labels to each value. If omitted, Pandas assigns default numeric indices (0, 1, 2, …).

Creating a DataFrame

A DataFrame can be created from a dictionary, list of dictionaries, or other data structures.

Code

# Creating a DataFrame from a dictionary
data = {
    'Name': ['Alice', 'Bob', 'Charlie'],
    'Age': [25, 30, 35],
    'City': ['New York', 'Los Angeles', 'Chicago']
}
df = pd.DataFrame(data)
print(df)

Output:

      Name  Age         City
0    Alice   25     New York
1      Bob   30  Los Angeles
2  Charlie   35      Chicago

Each key in the dictionary becomes a column, and the values form the rows.

3. Viewing Data

Pandas provides methods to inspect DataFrames.

View first few rows: df.head(n) (default n=5)
View last few rows: df.tail(n)
Get basic info: df.info()
Summary statistics: df.describe()

Code

# Viewing the first 2 rows
print(df.head(2))
# Get DataFrame info
print(df.info())

Selecting Data

You can select specific rows, columns, or values using indexing.

Selecting Columns

Access columns using their names.

Code

# Select a single column (returns a Series)
print(df['Name'])
# Select multiple columns (returns a DataFrame)
print(df[['Name', 'City']])

Selecting Rows

Use loc (label-based) or iloc (integer-based) to select rows.

Code

# Select row by index label
print(df.loc[0])
# Select row by integer position
print(df.iloc[1])

Filtering Data

Filter rows based on conditions.

Code

# Filter rows where Age > 30
print(df[df['Age'] > 30])

You can combine conditions using & (and) or | (or).

Code

# Filter rows where Age > 25 and City is Chicago
print(df[(df['Age'] > 25) & (df['City'] == 'Chicago')])

Adding and Modifying Data

You can add new columns or modify existing ones.

Adding a Column

Code

# Add a new column
df['Salary'] = [50000, 60000, 75000]
print(df)

Modifying Data

Update values using loc or conditions.

Code

# Update a specific value
df.loc[0, 'Age'] = 26
# Increase all salaries by 10%
df['Salary'] = df['Salary'] * 1.1
print(df)

Handling Missing Data

Pandas provides methods to handle missing values (NaN).

Code

# Introduce a missing value
df.loc[1, 'City'] = None
print(df)
# Fill missing values with a default
df['City'] = df['City'].fillna('Unknown')
print(df)

Grouping and Aggregating

Group data by a column and compute aggregates (e.g., mean, sum).

Code

# Group by City and calculate average Salary
print(df.groupby('City')['Salary'].mean())

aving and Loading Data

Pandas supports reading and writing data in various formats (CSV, Excel, JSON, etc.).

Saving to CSV

Code

df.to_csv('output.csv', index=False)

Loading from CSV

Code

df_from_csv = pd.read_csv('output.csv')
print(df_from_csv)

Advanced Pandas Features

Here are some advanced Pandas features to take your data analysis further.

Merging DataFrames

Pandas allows combining multiple DataFrames using merge or concat.

Merging DataFrames

Create a second DataFrame with additional information and merge it with the original.

Code

# Create a second DataFrame
data2 = {
    'Name': ['Alice', 'Bob', 'David'],
    'Department': ['HR', 'IT', 'Marketing']
}
df2 = pd.DataFrame(data2)

# Merge on 'Name' (inner join)
merged_df = pd.merge(df, df2, on='Name', how='inner')
print(merged_df)

The how parameter can be 'inner', 'left', 'right', or 'outer' to control which rows are kept.

Concatenating DataFrames

Use concat to stack DataFrames vertically or horizontally.

Code

# Concatenate vertically
df3 = pd.DataFrame({
    'Name': ['David'],
    'Age': [40],
    'City': ['Boston'],
    'Salary': [70000.0]
})
concat_df = pd.concat([df, df3], ignore_index=True)
print(concat_df)

Reshaping Data

Pandas provides tools like pivot, melt, and stack to reshape data.

Pivot Table

Create a pivot table to summarize data.

Code

# Add a 'Year' column for demonstration
df['Year'] = [2023, 2023, 2024]
pivot_table = df.pivot_table(values='Salary', index='City', columns='Year', aggfunc='mean')
print(pivot_table)

Melting Data

Convert wide-format data to long-format using melt.

Code

# Melt the DataFrame
melted_df = pd.melt(df, id_vars=['Name'], value_vars=['Age', 'Salary'], var_name='Metric', value_name='Value')
print(melted_df)

Time-Series Data

Pandas excels at handling time-series data.

Working with Dates

Convert a column to datetime and perform time-based operations.

Code

# Create a DataFrame with dates
df_time = pd.DataFrame({
    'Date': ['2023-01-01', '2023-02-01', '2023-03-01'],
    'Sales': [100, 150, 200]
})
df_time['Date'] = pd.to_datetime(df_time['Date'])

# Set Date as index
df_time.set_index('Date', inplace=True)

# Resample to monthly frequency and compute mean
monthly_sales = df_time.resample('ME').mean()
print(monthly_sales)

Rolling Windows

Calculate rolling statistics, like a moving average.

Code

# Calculate 2-period rolling mean
df_time['Rolling_Mean'] = df_time['Sales'].rolling(window=2).mean()
print(df_time)

Applying Custom Functions

Use apply or map to apply custom functions to DataFrames or Series.

Code

# Define a custom function
def categorize_salary(salary):
    return 'High' if salary > 70000 else 'Low'

# Apply to Salary column
df['Salary_Category'] = df['Salary'].apply(categorize_salary)
print(df)

Multi-Indexing

Multi-indexing allows hierarchical indexing for more complex data organization.

Creating a Multi-Index DataFrame

Set multiple columns as the index to create a multi-level index.

Code

# Set multi-index using City and Year
df_multi = df.set_index(['City', 'Year'])
print(df_multi)

Selecting with Multi-Index

Access data using multi-level indices with loc.

Code

# Select data for New York in 2023
print(df_multi.loc[('New York', 2023)])

Aggregating with Multi-Index

Group and aggregate with multi-level indices.

Code

# Group by City and Year, calculate average Salary
print(df_multi.groupby(level=['City', 'Year'])['Salary'].mean())

Handling Large Datasets with Chunking

For large datasets, Pandas can process data in chunks to manage memory usage.

Reading in Chunks

Use chunksize with read_csv to process large files incrementally.

Code

# Example: Reading a large CSV file in chunks
# Assume 'large_data.csv' exists
chunk_size = 2
for chunk in pd.read_csv('output.csv', chunksize=chunk_size):
    # Process each chunk (e.g., filter or aggregate)
    print(chunk.groupby('City')['Salary'].mean())

This approach is useful for files too large to fit in memory. You can perform operations (e.g., filtering, aggregating) on each chunk and combine results if needed.

Visualization with Matplotlib and Seaborn

Pandas integrates seamlessly with Matplotlib and Seaborn for data visualization.

Matplotlib: Bar Plot

Create a bar plot of average salaries by city.

Code

# Plot average salary by city
avg_salary = df.groupby('City')['Salary'].mean()
avg_salary.plot(kind='bar', title='Average Salary by City')
plt.xlabel('City')
plt.ylabel('Salary')
plt.show()

Output: A bar plot showing average salaries for each city.

Seaborn: Box Plot

Use Seaborn for a box plot to visualize salary distribution by salary category.

Code

# Box plot of salaries by Salary_Category
sns.boxplot(x='Salary_Category', y='Salary', data=df)
plt.title('Salary Distribution by Category')
plt.show()

Output: A box plot comparing salaries in ‘High’ and ‘Low’ categories.

Summary and Conclusion

This tutorial covers the essentials of Pandas for beginners, from creating and manipulating DataFrames and Series to advanced features like merging, reshaping, time-series analysis, multi-indexing, chunking for large datasets, and visualization with Matplotlib and Seaborn. Pandas is a versatile tool for data analysis, enabling efficient handling of structured data, complex indexing, and integration with visualization libraries. By mastering these concepts, you can perform sophisticated data manipulations and create insightful visualizations. Practice with real-world datasets to solidify your skills and explore Pandas’ full potential.

Resources

Official Pandas Documentation: https://pandas.pydata.org/docs/
Matplotlib Documentation: https://matplotlib.org/stable/contents.html
Seaborn Documentation: https://seaborn.pydata.org/
Kaggle Datasets: https://www.kaggle.com/datasets (for practice datasets)
Pandas Cheat Sheet: https://pandas.pydata.org/Pandas_Cheat_Sheet.pdf
Python Data Science Handbook by Jake VanderPlas (book or online resource for deeper learning)