Lesson · 40 min · Free
Unsupervised Domain Adaptation
Unsupervised Domain Adaptation body { font-family: sans-serif; line-height: 1.6; margin: 20px; } h1, h2 { color: #2c3e50; } pre, code { background-color: #ecf0f1; padding: 10px; border-radius: 5px; overflow-x: auto; } ul
Unsupervised Domain Adaptation
Welcome to this lesson on Unsupervised Domain Adaptation (UDA) within the context of AI in Drug Discovery. In the realm of machine learning, models are often trained on a specific dataset (the "source domain") and then expected to perform well on new, unseen data (the "target domain"). However, a common challenge arises when the statistical properties or distributions of the source and target domains differ significantly. This discrepancy, known as "domain shift," can lead to a drastic drop in model performance, even if the underlying task is the same. For example, a model trained to predict drug-target interactions using in vitro assay data might perform poorly when applied to in vivo animal study data, due to differences in noise levels, measurement techniques, or biological complexities. Similarly, a model trained on publicly available drug datasets might struggle with proprietary, internally generated data that has different chemical space coverage or assay biases. Unsupervised Domain Adaptation addresses this problem by attempting to adapt a model trained on a labeled source domain to an unlabeled target domain, without requiring any labels from the target domain itself. The core idea behind UDA is to learn a representation space where the features from both the source and target domains are indistinguishable or aligned, allowing a classifier trained on the source data to generalize effectively to the target data. This alignment can be achieved through various techniques, including adversarial learning, moment matching, or self-training. The "unsupervised" aspect is crucial in drug discovery, as obtaining extensive labeled data for every new experimental setup or patient cohort can be prohibitively expensive and time-consuming.
Why is Unsupervised Domain Adaptation Important in Drug Discovery?
UDA offers a powerful solution to several pervasive challenges in drug discovery: Data Scarcity: Labeled data for specific biological assays or clinical outcomes is often limited, especially for rare diseases or novel targets. UDA allows us to leverage existing, well-labeled datasets (source) and adapt models to new, unlabeled contexts (target). Batch Effects and Lab-to-Lab Variability: High-throughput screening data can suffer from batch effects, where measurements taken at different times or in different labs exhibit systematic differences. UDA can help mitigate these variations, making models more robust. Transition from in vitro to in vivo : Models trained on controlled in vitro experiments often face a significant domain shift when applied to complex in vivo or clinical data. UDA can bridge this gap. Proprietary vs. Public Data: Companies often have proprietary datasets that differ from publicly available ones. UDA enables the transfer of knowledge from public datasets to internal data without needing to re-label everything. Predictive Toxicology and ADMET: Predicting ADMET (Absorption, Distribution, Metabolism, Excretion, Toxicity) properties is critical. Models trained on one set of experimental conditions might need adaptation for new compounds or different biological systems. One popular approach to UDA involves adversarial training, inspired by Generative Adversarial Networks (GANs). In this setup, a feature extractor learns to produce features that are discriminative for the source task (e.g., classifying active vs. inactive compounds) but also indistinguishable to a domain discriminator. The domain discriminator, in turn, tries to distinguish between source and target domain features. This adversarial game forces the feature extractor to learn domain-invariant representations.
Code Example: Conceptual Adversarial Domain Adaptation (using PyTorch-like pseudocode)
This pseudocode illustrates the core components of an adversarial UDA setup. Note that this is a simplified representation for conceptual understanding. import torch import torch.nn as nn import torch.optim as optim # Assume we have preprocessed source_data (labeled) and target_data (unlabeled) # source_data_features, source_data_labels # target_data_features # 1. Feature Extractor (G) - Learns to extract features from input class FeatureExtractor(nn.Module): def __init__(self): super(FeatureExtractor, self).__init__() self.feature_layers = nn.Sequential( nn.Linear(input_dim, 128), nn.ReLU(), nn.Linear(128, 64), nn.ReLU() ) def forward(self, x): return self.feature_layers(x) # 2. Label Predictor (C) - Predicts labels based on extracted features (source task) class LabelPredictor(nn.Module): def __init__(self): super(LabelPredictor, self).__init__() self.classifier_layers = nn.Sequential( nn.Linear(64, 32), nn.ReLU(), nn.Linear(32, num_classes) # e.g., active/inactive ) def forward(self, x): return self.classifier_layers(x) # 3. Domain Discriminator (D) - Tries to distinguish source from target features class DomainDiscriminator(nn.Module): def __init__(self): super(DomainDiscriminator, self).__init__() self.discriminator_layers = nn.Sequential( nn.Linear(64, 32), nn.ReLU(), nn.Linear(32, 1), # Binary classification: source (0) or target (1) nn.Sigmoid() ) def forward(self, x): return self.discriminator_layers(x) # Initialize models feature_extractor = FeatureExtractor() label_predictor = LabelPredictor() domain_discriminator = DomainDiscriminator() # Optimizers optimizer_G_C = optim.Adam(list(feature_extractor.parameters()) + list(label_predictor.parameters()), lr=0.001) optimizer_D = optim.Adam(domain_discriminator.parameters(), lr=0.001) # Loss functions criterion_label = nn.CrossEntropyLoss() criterion_domain = nn.BCELoss() # Training loop (simplified) num_epochs = 100 for epoch in range(num_epochs): # --- Train Discriminator --- optimizer_D.zero_grad() # Source features features_source = feature_extractor(source_data_features.detach()) # Detach to stop gradient flow to G domain_pred_source = domain_discriminator(features_source) loss_D_source = criterion_domain(domain_pred_source, torch.zeros_like(domain_pred_source)) # Label as 0 (source) # Target features features_target = feature_extractor(target_data_features.detach()) # Detach domain_pred_target = domain_discriminator(features_target) loss_D_target = criterion_domain(domain_pred_target, torch.ones_like(domain_pred_target)) # Label as 1 (target) loss_D = loss_D_source + loss_D_target loss_D.backward() optimizer_D.step() # --- Train Feature Extractor and Label Predictor --- optimizer_G_C.zero_grad() # Source task loss features_source_G = feature_extractor(source_data_features) label_pred = label_predictor(features_source_G) loss_label = criterion_label(label_pred, source_data_labels) # Domain adaptation loss (G tries to fool D) features_target_G = feature_extractor(target_data_features) domain_pred_target_G = domain_discriminator(features_target_G) loss_domain_G = criterion_domain(domain_pred_target_G, torch.zeros_like(domain_pred_target_G)) # G wants D to think target is source # Total loss for G and C # lambda_tradeoff controls the balance between task loss and domain alignment loss lambda_tradeoff = 0.1 total_loss_G_C = loss_label + lambda_tradeoff * loss_domain_G total_loss_G_C.backward() optimizer_G_C.step() # Print losses for monitoring if (epoch+1) % 10 == 0: print(f"Epoch [{epoch+1}/{num_epochs}], Loss_D: {loss_D.item():.4f}, Loss_G_C: {total_loss_G_C.item():.4f}, Loss_Label: {loss_label.item():.4f}") # After training, the label_predictor can be used on target_data_features # to make predictions, as feature_extractor has learned domain-invariant features. Another technique is Maximum Mean Discrepancy (MMD), which directly measures the distance between the mean embeddings of features from two domains in a Reproducing Kernel Hilbert Space (RKHS). The goal is to minimize this distance during training, thereby aligning the distributions. This is often combined with a standard classification loss on the source data.
Code Example: Conceptual MMD-based Domain Adaptation (using PyTorch-like pseudocode)
import torch import torch.nn as nn import torch.optim as optim # Assume we have preprocessed source_data (labeled) and target_data (unlabeled) # source_data_features, source_data_labels # target_data_features # Feature Extractor (G) class FeatureExtractor(nn.Module): def __init__(self): super(FeatureExtractor, self).__init__() self.feature_layers = nn.Sequential( nn.Linear(input_dim, 128), nn.ReLU(), nn.Linear(128, 64), nn.ReLU() ) def forward(self, x): return self.feature_layers(x) # Label Predictor (C) class LabelPredictor(nn.Module): def __init__(self): super(LabelPredictor, self).__init__() self.classifier_layers = nn.Sequential( nn.Linear(64, 32), nn.ReLU(), nn.Linear(32, num_classes) ) def forward(self, x): return self.classifier_layers(x) # MMD Loss function (simplified RBF kernel) def mmd_rbf_loss(x, y, gamma=1.0): """ Calculates the Maximum Mean Discrepancy (MMD) using a radial basis function (RBF) kernel. x: features from domain 1 y: features from domain 2 """ xx = torch.matmul(x, x.transpose(0, 1)) yy = torch.matmul(y, y.transpose(0, 1)) xy = torch.matmul(x, y.transpose(0, 1)) rx = (xx.diag().unsqueeze(0).expand_as(xx)) ry = (yy.diag().unsqueeze(0).expand_as(yy)) dxx = rx.transpose(0, 1) + rx - 2.*xx dyy = ry.transpose(0, 1) + ry - 2.*yy dxy = rx.transpose(0, 1) + ry - 2.*xy K_xx = torch.exp(-gamma * dxx) K_yy = torch.exp(-gamma * dyy) K_xy = torch.exp(-gamma * dxy) # MMD^2 = 1/n^2 sum(K_xx) + 1/m^2 sum(K_yy) - 2/(nm) sum(K_xy) mmd_loss = torch.mean(K_xx) + torch.mean(K_yy) - 2 * torch.mean(K_
Watch the full lesson — free
This topic is part of AI in Drug Discovery, a complete AI-narrated video course. Press play once and watch the entire lecture like a movie.
Start the course free →