Lesson · 40 min · Free
Pandas DataFrames Masterclass
Pandas DataFrames Masterclass body { font-family: sans-serif; line-height: 1.6; color: #333; margin: 20px; } h1, h2 { color: #2C3E50; } pre { background-color: #ECF0F1; padding: 15px; border-radius: 5px; overflow-x: auto
Pandas DataFrames Masterclass
Welcome to the Pandas DataFrames Masterclass, a crucial component of your "Python for Data Science" journey. For pharmacy and biotech professionals, the ability to efficiently manage and analyze experimental data, patient records, or drug trial results is paramount. Pandas DataFrames provide a powerful, flexible, and intuitive way to handle tabular data in Python, making them an indispensable tool in your analytical toolkit. This lesson will equip you with the fundamental concepts and practical skills to confidently work with DataFrames, from creation to advanced manipulation. At its core, a Pandas DataFrame can be thought of as a two-dimensional, size-mutable, and potentially heterogeneous tabular data structure with labeled axes (rows and columns). Imagine it as a spreadsheet or a SQL table. Each column in a DataFrame is a Pandas Series, which is a one-dimensional labeled array capable of holding any data type. This structure allows for powerful indexing, slicing, and arithmetic operations across both rows and columns, facilitating complex data transformations with relative ease.
Creating and Inspecting DataFrames
Let's begin by understanding how to create DataFrames. The most common methods involve using dictionaries of lists, lists of dictionaries, or reading from external files like CSVs or Excel spreadsheets. For our purposes, we'll focus on in-memory creation initially, as this helps to build an understanding of the DataFrame's internal structure. Consider a scenario where you have collected preliminary data from a small-scale drug efficacy study. You might have information on patient IDs, their assigned treatment group (e.g., 'Placebo', 'Drug A', 'Drug B'), baseline biomarker levels, and a post-treatment response score. This data can be easily represented as a DataFrame. import pandas as pd import numpy as np # Data for a hypothetical drug efficacy study data = { 'PatientID': ['P001', 'P002', 'P003', 'P004', 'P005', 'P006'], 'TreatmentGroup': ['Placebo', 'Drug A', 'Drug B', 'Placebo', 'Drug A', 'Drug B'], 'BaselineBiomarker': [25.3, 31.8, 28.1, 26.5, 30.2, 29.7], 'ResponseScore': [1.2, 4.5, 3.8, 1.5, 4.1, 3.5], 'Age': [45, 52, 60, 48, 55, 63] } # Create the DataFrame df_study = pd.DataFrame(data) print("--- Initial DataFrame ---") print(df_study) print("\n--- DataFrame Info ---") df_study.info() print("\n--- DataFrame Head (first 3 rows) ---") print(df_study.head(3)) print("\n--- DataFrame Describe (numerical columns) ---") print(df_study.describe()) The .info() method provides a concise summary of the DataFrame, including the data types of each column and the number of non-null values, which is crucial for identifying missing data. .head() and .tail() allow for quick inspection of the first or last few rows, respectively. The .describe() method generates descriptive statistics for numerical columns, offering insights into central tendency, dispersion, and shape of the distribution, which is highly useful for initial data exploration in pharmaceutical research.
Indexing and Selection
Accessing specific subsets of your data is a fundamental operation. Pandas offers several powerful methods for indexing and selecting data: df[...] : Column selection (returns a Series if single column, DataFrame if multiple). .loc[] : Label-based indexing for rows and columns. .iloc[] : Integer-location based indexing for rows and columns. Boolean indexing: Selecting rows based on conditions. Let's illustrate these with our drug study DataFrame. Suppose we want to analyze only the patients in 'Drug A' group, or retrieve the 'BaselineBiomarker' for a specific patient ID. # Select a single column baseline_biomarkers = df_study['BaselineBiomarker'] print("\n--- Baseline Biomarkers Series ---") print(baseline_biomarkers) # Select multiple columns treatment_and_response = df_study[['TreatmentGroup', 'ResponseScore']] print("\n--- Treatment Group and Response Score DataFrame ---") print(treatment_and_response) # Select row(s) by label using .loc[] # Here, we'll set 'PatientID' as the index for easier label-based row selection df_study_indexed = df_study.set_index('PatientID') patient_P003_data = df_study_indexed.loc['P003'] print("\n--- Data for Patient P003 (using .loc) ---") print(patient_P003_data) # Select rows and specific columns by label drug_a_baseline_age = df_study_indexed.loc[df_study_indexed['TreatmentGroup'] == 'Drug A', ['BaselineBiomarker', 'Age']] print("\n--- Baseline Biomarker and Age for 'Drug A' patients (using .loc with boolean indexing) ---") print(drug_a_baseline_age) # Select row(s) by integer position using .iloc[] second_patient_data = df_study.iloc[1] # Note: df_study here is not indexed by PatientID print("\n--- Data for the second patient (using .iloc) ---") print(second_patient_data) # Select specific cell by integer position (row 0, column 2 - BaselineBiomarker) first_patient_baseline = df_study.iloc[0, 2] print(f"\n--- Baseline Biomarker for the first patient (using .iloc): {first_patient_baseline} ---") # Boolean indexing: Patients with ResponseScore > 3.0 high_responders = df_study[df_study['ResponseScore'] > 3.0] print("\n--- Patients with ResponseScore > 3.0 (using boolean indexing) ---") print(high_responders) Understanding the distinction between .loc[] and .iloc[] is critical. .loc[] uses the actual labels of your index and columns, which are often more descriptive (e.g., 'PatientID', 'TreatmentGroup'). In contrast, .iloc[] uses the integer positions (0-based) of rows and columns, similar to standard Python list indexing. Boolean indexing, on the other hand, allows for powerful conditional selection, enabling you to filter your data based on specific criteria, such as selecting all patients above a certain age or those exhibiting a particular adverse event.
Key Takeaways
Pandas DataFrames are 2D, labeled data structures, analogous to spreadsheets, ideal for tabular data in pharmacy/biotech. DataFrames are composed of Pandas Series (columns), each capable of holding different data types. Common creation methods include dictionaries of lists or reading from files (CSV, Excel). .info() , .head() , .tail() , and .describe() are essential for initial data inspection and understanding. df[...] , .loc[] , .iloc[] , and boolean indexing provide flexible and powerful ways to select and filter data. .loc[] uses labels (e.g., 'PatientID', 'TreatmentGroup'), while .iloc[] uses integer positions (0, 1, 2...).
Practice Exercise: Adverse Event Analysis
Imagine you have a DataFrame representing adverse events (AEs) reported during a clinical trial. Each row corresponds to a single AE report. Your task is to perform the following operations: Create a Pandas DataFrame from the following dictionary: ae_data = { 'ReportID': [101, 102, 103, 104, 105, 106, 107, 108], 'PatientID': ['P001', 'P002', 'P001', 'P003', 'P004', 'P002', 'P005', 'P003'], 'AE_Term': ['Nausea', 'Headache', 'Dizziness', 'Nausea', 'Fatigue', 'Vomiting', 'Headache', 'Rash'], 'Severity': ['Mild', 'Moderate', 'Mild', 'Moderate', 'Severe', 'Moderate', 'Mild', 'Mild'], 'TreatmentGroup': ['Drug A', 'Placebo', 'Drug A', 'Drug B', 'Drug A', 'Placebo', 'Drug B', 'Drug B'], 'DaysPostTreatment': [5, 10, 7, 8, 12, 11, 6, 9] } Display the first 4 rows of your DataFrame. Show a summary of the DataFrame's information (data types, non-null counts). Select and display only the 'AE_Term' and 'Severity' columns for all reports. Using boolean indexing, filter and display all adverse events reported by patients in the 'Drug A' treatment group. Using .loc[] , retrieve all data for the adverse event with 'ReportID' 105. (Hint: you might need to set 'ReportID' as your index first).
Watch the full lesson — free
This topic is part of Python for Data Science, a complete AI-narrated video course. Press play once and watch the entire lecture like a movie.
Start the course free →