Lesson · 40 min · Free
Pandas DataFrames Essentials
Pandas DataFrames Essentials body { font-family: sans-serif; line-height: 1.6; margin: 20px; } h1, h2 { color: #2c3e50; } pre { background-color: #ecf0f1; padding: 15px; border-radius: 5px; overflow-x: auto; } code { fon
Pandas DataFrames Essentials
Welcome to the "Pandas DataFrames Essentials" lesson, a crucial component of your "Python for Data Science" journey. As aspiring professionals in pharmacy and biotechnology, you will frequently encounter datasets that require robust manipulation and analysis. Pandas DataFrames are the workhorse for such tasks in Python, offering a powerful and flexible two-dimensional, size-mutable, and potentially heterogeneous tabular data structure with labeled axes (rows and columns). Think of a DataFrame as a sophisticated spreadsheet or a SQL table, but with far greater programmatic capabilities. Understanding DataFrames is fundamental. They are built upon NumPy arrays, inheriting their efficiency, but add labeled indexing for both rows and columns, making data much more intuitive to work with. This labeling is particularly beneficial when dealing with experimental results, patient demographics, drug efficacy trials, or genomic sequencing data, where specific identifiers and meaningful column names are paramount. In this lesson, we will cover the basics of creating, inspecting, and performing elementary operations on DataFrames, laying the groundwork for more advanced data analysis.
Creating and Inspecting DataFrames
DataFrames can be created from various data sources, including dictionaries, lists of lists, NumPy arrays, or by reading external files like CSV, Excel, or SQL databases. For our initial exploration, we'll focus on creating them from Python dictionaries, which allows us to explicitly define column names and their corresponding data. Once created, inspecting the DataFrame's structure and content is a primary step in understanding your data. import pandas as pd import numpy as np # Example 1: Creating a DataFrame from a dictionary # This simulates patient data from a clinical trial patient_data = { 'Patient_ID': ['P001', 'P002', 'P003', 'P004', 'P005'], 'Age': [45, 62, 38, 55, 71], 'Gender': ['Male', 'Female', 'Female', 'Male', 'Female'], 'Drug_Dosage_mg': [100, 150, 100, 200, 150], 'Response_Score': [7.2, 8.5, 6.9, 9.1, 7.8], # e.g., on a scale of 1-10 'Adverse_Event': [False, True, False, False, True] } df_patients = pd.DataFrame(patient_data) print("--- Initial DataFrame ---") print(df_patients) print("\n--- DataFrame Info (data types, non-null counts) ---") df_patients.info() print("\n--- First 3 rows of the DataFrame ---") print(df_patients.head(3)) print("\n--- Descriptive Statistics for Numerical Columns ---") print(df_patients.describe()) The .info() method is invaluable for quickly understanding the data types of each column (e.g., int64 , float64 , bool , object for strings) and the number of non-null entries, which helps in identifying missing data. .head() and .tail() allow you to peek at the beginning or end of your DataFrame, respectively, while .describe() provides a statistical summary of numerical columns, including count, mean, standard deviation, min/max, and quartiles – metrics highly relevant in clinical and experimental analysis.
Basic DataFrame Operations: Selection and Filtering
Once you have a DataFrame, the next step is often to select specific columns or rows, or to filter data based on certain conditions. This is where the power of DataFrames truly shines, allowing for intuitive and efficient data subsetting. You can select single columns, multiple columns, or use boolean indexing to filter rows that meet specific criteria, which is essential for identifying patient cohorts, specific experimental conditions, or adverse event occurrences. # Example 2: Selecting and Filtering Data # Select a single column (returns a Series) ages = df_patients['Age'] print("\n--- Patient Ages (Series) ---") print(ages.head()) # Select multiple columns (returns a DataFrame) patient_identifiers_and_response = df_patients[['Patient_ID', 'Response_Score']] print("\n--- Patient IDs and Response Scores (DataFrame) ---") print(patient_identifiers_and_response.head()) # Filter rows based on a condition (e.g., patients with high response scores) high_responders = df_patients[df_patients['Response_Score'] > 8.0] print("\n--- Patients with Response Score > 8.0 ---") print(high_responders) # Filter for female patients who experienced an adverse event adverse_female_patients = df_patients[(df_patients['Gender'] == 'Female') & (df_patients['Adverse_Event'] == True)] print("\n--- Female Patients with Adverse Events ---") print(adverse_female_patients) # Using .loc for label-based indexing (e.g., selecting rows by index label, and specific columns) # Here we're selecting the first two rows (index 0 and 1) and 'Age', 'Drug_Dosage_mg' columns subset_loc = df_patients.loc[0:1, ['Age', 'Drug_Dosage_mg']] print("\n--- Subset using .loc (rows 0-1, cols 'Age', 'Drug_Dosage_mg') ---") print(subset_loc) # Using .iloc for integer-location based indexing (e.g., selecting rows by integer position, and specific columns) # Here we're selecting the first three rows (index 0, 1, 2) and the first two columns (index 0, 1) subset_iloc = df_patients.iloc[0:3, 0:2] print("\n--- Subset using .iloc (rows 0-2, cols 0-1) ---") print(subset_iloc) The methods .loc and .iloc are crucial for more precise selection. .loc is primarily label-based, meaning you use the actual row and column names for selection. In contrast, .iloc is integer-location based, using the numerical positions of rows and columns. Mastering these indexing techniques is vital for efficient and unambiguous data access, especially when dealing with complex experimental designs or large-scale clinical trial data where specific subsets need to be isolated for further analysis.
Key Takeaways
Pandas DataFrames are tabular, labeled data structures, ideal for structured datasets in pharmacy and biotech. They can be created from various sources, including dictionaries, and offer powerful inspection methods like .info() , .head() , and .describe() . Individual columns can be selected using bracket notation ( df['ColumnName'] ), returning a Pandas Series. Multiple columns are selected using a list of column names ( df[['Col1', 'Col2']] ), returning a DataFrame. Rows can be filtered using boolean indexing based on conditions (e.g., df[df['Score'] > 8.0] ). .loc is used for label-based indexing (row/column names), while .iloc is used for integer-location based indexing (row/column positions).
Practice Exercise: Clinical Biomarker Analysis
Imagine you're analyzing biomarker data from a study on a new therapeutic agent. Create a Pandas DataFrame named biomarker_data with the following columns and at least 5 rows of made-up data: Patient_ID (e.g., 'P101', 'P102') Treatment_Group (e.g., 'Placebo', 'Low Dose', 'High Dose') Biomarker_A_Level (numerical, e.g., 25.3, 30.1) Biomarker_B_Level (numerical, e.g., 120.5, 115.9) Adverse_Reaction (boolean, True/False) Then, perform the following operations: Print the first 3 rows of your DataFrame. Display the data types and non-null counts for all columns. Select and print only the Patient_ID and Biomarker_A_Level columns. Filter and print all patients who are in the 'High Dose' group AND have experienced an Adverse_Reaction . Using .loc , select the Biomarker_A_Level and Biomarker_B_Level for the first two patients (based on their row index labels).
Watch the full lesson — free
This topic is part of Python for Data Science, a complete AI-narrated video course. Press play once and watch the entire lecture like a movie.
Start the course free →