Lesson · 40 min · Free
K-Means on Real Data
K-Means on Real Data K-Means on Real Data Welcome to this lesson on applying the K-Means clustering algorithm to real-world datasets. In the realm of pharmacy and biotechnology, unsupervised learning techniques like K-Me
K-Means on Real Data
Welcome to this lesson on applying the K-Means clustering algorithm to real-world datasets. In the realm of pharmacy and biotechnology, unsupervised learning techniques like K-Means are invaluable for uncovering hidden patterns and structures within complex biological and clinical data. For instance, K-Means can be used to identify patient subgroups with similar disease progression profiles, cluster drug compounds based on their molecular properties, or categorize gene expression data to reveal novel biological pathways. While the theoretical underpinnings of K-Means involve minimizing the sum of squared distances between data points and their assigned cluster centroids, its practical application requires careful data preparation and interpretation. Real data often presents challenges such as varying scales, missing values, and the need for appropriate feature selection. Our goal in this lesson is to demonstrate a robust workflow for applying K-Means, focusing on techniques relevant to your domain. We will be using Python's powerful libraries, scikit-learn for the K-Means implementation and pandas for data manipulation, along with matplotlib for visualization. Let's start by simulating a dataset that mimics a common scenario in biotech: gene expression levels across different samples. Imagine we have expression data for several genes, and we want to group samples that exhibit similar gene expression patterns.
Implementing K-Means with Scikit-learn
Our first step is to generate or load our data. For demonstration purposes, we'll create a synthetic dataset that represents gene expression profiles. We'll then scale the data, which is crucial for K-Means as it is sensitive to the magnitude of features. Without scaling, features with larger values might disproportionately influence the clustering results. import pandas as pd import numpy as np from sklearn.cluster import KMeans from sklearn.preprocessing import StandardScaler import matplotlib.pyplot as plt import seaborn as sns # Simulate a dataset: Gene expression data for 100 samples and 5 genes # Imagine three underlying groups of samples np.random.seed(42) # for reproducibility # Group 1: Low expression for genes 1, 2; high for 3, 4, 5 group1 = np.random.normal(loc=[2, 2, 8, 8, 8], scale=1.5, size=(30, 5)) # Group 2: Medium expression across all genes group2 = np.random.normal(loc=[5, 5, 5, 5, 5], scale=1.5, size=(40, 5)) # Group 3: High expression for genes 1, 2; low for 3, 4, 5 group3 = np.random.normal(loc=[8, 8, 2, 2, 2], scale=1.5, size=(30, 5)) # Combine groups into a single dataset gene_expression_data = np.vstack((group1, group2, group3)) df = pd.DataFrame(gene_expression_data, columns=[f'Gene_{i+1}' for i in range(5)]) print("Original Data Head:") print(df.head()) print("\nOriginal Data Description:") print(df.describe()) # Step 1: Data Scaling # K-Means is distance-based, so scaling is crucial. scaler = StandardScaler() scaled_data = scaler.fit_transform(df) scaled_df = pd.DataFrame(scaled_data, columns=df.columns) print("\nScaled Data Head:") print(scaled_df.head()) # Step 2: Apply K-Means # We need to choose the number of clusters, 'k'. # For this example, let's assume we expect 3 groups. n_clusters = 3 kmeans = KMeans(n_clusters=n_clusters, random_state=42, n_init=10) # n_init for robustness kmeans.fit(scaled_data) # Get cluster assignments df['Cluster'] = kmeans.labels_ print("\nData with Cluster Assignments Head:") print(df.head()) # Step 3: Analyze Cluster Centroids # The centroids represent the 'average' profile of each cluster. cluster_centroids = pd.DataFrame(scaler.inverse_transform(kmeans.cluster_centers_), columns=df.columns[:-1]) print("\nCluster Centroids (unscaled values):") print(cluster_centroids) # Step 4: Visualize the results (example using a pair plot for a subset of genes) # This helps in understanding the separation of clusters. sns.pairplot(df, hue='Cluster', vars=['Gene_1', 'Gene_2', 'Gene_3']) plt.suptitle('Pair Plot of Gene Expression by Cluster', y=1.02) plt.show() # You can also visualize the centroids directly cluster_centroids.plot(kind='bar', figsize=(10, 6)) plt.title('Gene Expression Levels for Each Cluster Centroid') plt.ylabel('Expression Level') plt.xlabel('Gene') plt.xticks(rotation=45) plt.legend(title='Cluster') plt.tight_layout() plt.show() In the code above, we first generate our synthetic gene expression data. We then apply StandardScaler to ensure all genes contribute equally to the distance calculations. After scaling, we initialize and fit the KMeans model. The n_clusters parameter is critical and often determined through methods like the elbow method or domain knowledge. Finally, we add the cluster labels back to our original DataFrame and visualize the results. Visualizing the cluster centroids, especially after inverse-transforming them back to the original scale, provides intuitive insights into the characteristics of each identified group. Choosing the optimal number of clusters ( k ) is a common challenge. While we assumed k=3 in the previous example, in a real-world scenario, you might not know the true number of underlying groups. The "Elbow Method" is a heuristic often used to estimate k by looking for a bend in the plot of the sum of squared distances (inertia) against the number of clusters. Let's demonstrate this: # Elbow Method to find optimal K inertia = [] range_k = range(1, 11) # Test k from 1 to 10 for k in range_k: kmeans_model = KMeans(n_clusters=k, random_state=42, n_init=10) kmeans_model.fit(scaled_data) inertia.append(kmeans_model.inertia_) # inertia is the sum of squared distances plt.figure(figsize=(8, 5)) plt.plot(range_k, inertia, marker='o') plt.title('Elbow Method for Optimal K') plt.xlabel('Number of Clusters (K)') plt.ylabel('Inertia (Sum of Squared Distances)') plt.grid(True) plt.show() print("\nInertia values for different K:") for k_val, inertia_val in zip(range_k, inertia): print(f"K={k_val}: Inertia={inertia_val:.2f}") In the elbow plot, you look for the "elbow point" where the rate of decrease in inertia significantly slows down. This point often suggests a reasonable number of clusters. For our simulated data, you would likely observe a clear elbow at K=3, reinforcing our initial assumption.
Key Takeaways
K-Means is a powerful unsupervised learning algorithm for grouping similar data points. Data scaling (e.g., using StandardScaler ) is crucial for K-Means, especially with features on different scales. The number of clusters, k , must be chosen carefully, often using methods like the Elbow Method or domain expertise. Analyzing cluster centroids provides valuable insights into the characteristics of each identified group. Visualization is essential for interpreting K-Means results and understanding cluster separation.
Practice Exercise
Imagine you have a dataset of patient vital signs (e.g., heart rate, blood pressure, temperature, oxygen saturation) collected over time. Your task is to use K-Means to identify distinct patient subgroups based on these vital signs. Create a synthetic dataset for 150 patients with 4 vital sign features. Introduce at least two distinct underlying patient groups (e.g., "stable" and "at-risk"). Apply StandardScaler to normalize the vital sign data. Use the Elbow Method to determine an appropriate number of clusters (K). Perform K-Means clustering with your chosen K. Analyze and interpret the cluster centroids. What do these centroids tell you about the characteristics of each patient subgroup? Visualize the clusters using a pair plot or a scatter plot of two key vital signs, colored by cluster assignment. Focus on the practical steps and the interpretation of the results in a clinical context.
Watch the full lesson — free
This topic is part of Python Programming - Basics, a complete AI-narrated video course. Press play once and watch the entire lecture like a movie.
Start the course free →