Lesson · 40 min · Free
EDA in Python
EDA in Python body { font-family: sans-serif; line-height: 1.6; color: #333; } h1, h2 { color: #0056b3; } pre { background-color: #f4f4f4; border: 1px solid #ddd; padding: 10px; overflow-x: auto; font-family: monospace;
EDA in Python
Welcome to this lesson on Exploratory Data Analysis (EDA) in Python. As future professionals in pharmacy and biotech, you'll frequently encounter datasets ranging from clinical trial results to genomic sequencing data. Before diving into complex statistical models or machine learning algorithms, it's crucial to understand your data's underlying structure, identify patterns, detect anomalies, and test hypotheses. This initial investigation is precisely what EDA is about. EDA is a critical first step in any data analysis project. It's an iterative process of inspecting, cleaning, and transforming data with the goal of discovering insights, summarizing its main characteristics, and often visualizing those characteristics. Think of it as getting to know your dataset intimately before you ask it to answer complex questions. For pharmacy and biotech applications, this could mean understanding patient demographics, drug efficacy distributions, gene expression levels, or protein interaction networks.
Key Steps and Tools for EDA
In Python, the primary libraries for EDA are pandas for data manipulation and matplotlib and seaborn for data visualization. We'll focus on using these tools to perform common EDA tasks.
1. Data Loading and Initial Inspection
The first step is always to load your data into a pandas DataFrame and get a high-level overview. This involves checking data types, missing values, and basic descriptive statistics. import pandas as pd # Let's imagine we have a CSV file named 'clinical_trial_data.csv' # containing patient demographics, drug dosages, and efficacy outcomes. try: df = pd.read_csv('clinical_trial_data.csv') except FileNotFoundError: print("clinical_trial_data.csv not found. Creating a sample DataFrame for demonstration.") data = { 'PatientID': range(1, 11), 'Age': [28, 35, 42, 55, 61, 30, 48, 52, 65, 39], 'Gender': ['M', 'F', 'F', 'M', 'F', 'M', 'F', 'M', 'F', 'M'], 'DrugDose_mg': [10, 15, 10, 20, 15, 10, 20, 15, 20, 10], 'EfficacyScore': [7.2, 8.5, 6.8, 9.1, 7.5, 7.0, 8.8, 7.9, 9.5, 7.3], 'AdverseEvent': ['None', 'Nausea', 'None', 'Headache', 'None', 'None', 'Rash', 'None', 'Nausea', 'None'] } df = pd.DataFrame(data) print("--- First 5 rows of the DataFrame ---") print(df.head()) print("\n--- DataFrame Information (data types, non-null counts) ---") print(df.info()) print("\n--- Descriptive Statistics for Numerical Columns ---") print(df.describe()) The .head() method shows the first few rows, giving you a quick glance at the data. .info() is invaluable for understanding data types and identifying columns with missing values. .describe() provides summary statistics (mean, standard deviation, min, max, quartiles) for numerical columns, which is excellent for understanding the distribution and potential outliers.
2. Handling Missing Values
Missing data is common in real-world datasets, especially in biological and clinical research. Identifying and appropriately handling missing values is a crucial EDA step. Strategies include imputation (filling with mean, median, mode) or removal of rows/columns, depending on the extent and nature of the missingness. # Let's intentionally introduce some missing values for demonstration import numpy as np df_missing = df.copy() df_missing.loc[2, 'EfficacyScore'] = np.nan df_missing.loc[7, 'DrugDose_mg'] = np.nan df_missing.loc[0, 'AdverseEvent'] = np.nan print("\n--- Missing values before handling ---") print(df_missing.isnull().sum()) # Example: Filling missing 'EfficacyScore' with the median median_efficacy = df_missing['EfficacyScore'].median() df_missing['EfficacyScore'].fillna(median_efficacy, inplace=True) # Example: Dropping rows where 'DrugDose_mg' is missing df_missing.dropna(subset=['DrugDose_mg'], inplace=True) # Example: Filling missing 'AdverseEvent' with 'Unknown' df_missing['AdverseEvent'].fillna('Unknown', inplace=True) print("\n--- Missing values after handling ---") print(df_missing.isnull().sum()) print("\n--- DataFrame after handling missing values (first 5 rows) ---") print(df_missing.head()) The choice of how to handle missing data depends heavily on the context. For clinical data, simply imputing with a mean might obscure important variability, so careful consideration is needed.
3. Data Visualization
Visualization is perhaps the most powerful aspect of EDA. It allows us to quickly grasp distributions, relationships between variables, and identify outliers that might be hard to spot in raw numbers. import matplotlib.pyplot as plt import seaborn as sns # Set a style for plots sns.set_style("whitegrid") # Histogram of Age plt.figure(figsize=(8, 5)) sns.histplot(df['Age'], kde=True, bins=5, color='skyblue') plt.title('Distribution of Patient Age') plt.xlabel('Age (Years)') plt.ylabel('Count') plt.show() # Box plot of EfficacyScore by Gender plt.figure(figsize=(8, 5)) sns.boxplot(x='Gender', y='EfficacyScore', data=df, palette='pastel') plt.title('Efficacy Score Distribution by Gender') plt.xlabel('Gender') plt.ylabel('Efficacy Score') plt.show() # Scatter plot of Age vs. EfficacyScore, colored by DrugDose plt.figure(figsize=(10, 6)) sns.scatterplot(x='Age', y='EfficacyScore', hue='DrugDose_mg', size='DrugDose_mg', data=df, palette='viridis', sizes=(50, 200)) plt.title('Age vs. Efficacy Score by Drug Dose') plt.xlabel('Age (Years)') plt.ylabel('Efficacy Score') plt.legend(title='Drug Dose (mg)') plt.show() # Count plot of Adverse Events plt.figure(figsize=(9, 5)) sns.countplot(y='AdverseEvent', data=df, order=df['AdverseEvent'].value_counts().index, palette='coolwarm') plt.title('Frequency of Adverse Events') plt.xlabel('Count') plt.ylabel('Adverse Event') plt.show() These plots quickly reveal insights: the age distribution of patients, potential differences in drug efficacy between genders, the relationship between age, drug dose, and efficacy, and the most common adverse events. For instance, a box plot could show if a new drug has a significantly different efficacy profile in males vs. females, or a scatter plot might reveal a dose-response relationship.
4. Correlation Analysis
Understanding the relationships between numerical variables is important. Correlation matrices and heatmaps can quickly highlight strong positive or negative correlations. # Select only numerical columns for correlation calculation numerical_df = df.select_dtypes(include=[np.number]) print("\n--- Correlation Matrix ---") print(numerical_df.corr()) plt.figure(figsize=(8, 6)) sns.heatmap(numerical_df.corr(), annot=True, cmap='coolwarm', fmt=".2f") plt.title('Correlation Matrix of Numerical Features') plt.show() A heatmap makes it easy to spot strong correlations, e.g., if higher drug doses correlate with higher efficacy scores, or if age is inversely correlated with certain biomarkers. EDA is an iterative process. You might perform an initial visualization, identify an outlier, investigate its cause (e.g., data entry error or a true biological anomaly), clean the data, and then re-visualize. This continuous exploration helps you build a robust understanding of your data before moving to more advanced analysis.
Key Takeaways:
EDA is foundational: It's the first and most crucial step in any data analysis project, especially in complex pharmacy/biotech data. pandas for data manipulation: Essential for loading, inspecting, cleaning, and transforming your datasets. matplotlib and seaborn for visualization: Powerful tools for creating informative plots to reveal patterns, distributions, and relationships. Understand your data types: Incorrect data types can lead to errors and misinterpretations. Address missing values wisely: The strategy for handling missing data should be context-dependent. Look for outliers and anomalies: These can be errors or genuine, important biological events. EDA is iterative: It's a continuous cycle of questioning, exploring, and refining your understanding of the data.
Practice Exercise: Analyzing a Hypothetical Gene Expression Dataset
Imagine you have a dataset representing gene expression levels (mRNA counts) in two different cell lines ( CellLine_A , CellLine_B ) for 100 genes, along with a 'Treatment_Group' ( Control or Treated ) and 'Gene_Function_Category' (e.g., Metabolism , Apoptosis , Signaling ). Your task is to perform an initial EDA using Python. Steps: Create a synthetic DataFrame: Generate a pandas DataFrame with columns for 'GeneID', 'CellLine_A_Expression', 'CellLine_B_Expression', 'Treatment_Group', and 'Gene_Function_Category'. Populate it with 100 rows of suitable dummy data (e.g., random numbers for expression, random choices for groups/categories). Initial Inspection: Use .head() , .info() , and .describe() to get a first look at your simulated data. Check for Missing Values: Introduce 5-10 random missing values in your expression columns and then use .isnull().sum() to identify them. Decide on a simple strategy (e.g., fill with median) and apply it. Visualize Distributions: Create histograms for 'CellLine_A_Expression' and 'CellLine_B_Expression'. Use box plots to compare 'CellLine_A_Expression' between 'Control' and 'Treated' groups. Create a count plot for 'Gene_Function_Category'. Analyze Relationships: Create a scatter plot of 'CellLine_A_Expression' vs. 'CellLine_B_Expression', coloring points by 'Treatment_Group'. What might this tell you about gene expression consistency between cell lines under different treatments? Calculate and visualize the correlation matrix for your numerical expression columns. This exercise will reinforce your understanding of core EDA techniques using
Watch the full lesson — free
This topic is part of Python for Data Science, a complete AI-narrated video course. Press play once and watch the entire lecture like a movie.
Start the course free →