Lesson · 40 min · Free
Multimodal AI: Text, Images, Audio & Video
Multimodal AI: Text, Images, Audio & Video body { font-family: sans-serif; line-height: 1.6; margin: 20px; } h1, h2 { color: #2c3e50; } pre { background-color: #ecf0f1; padding: 15px; border-radius: 5px; overflow-x: auto
Multimodal AI: Text, Images, Audio & Video
Welcome to this lesson on Multimodal AI , a crucial frontier in generative AI. As future innovators in pharmacy and biotechnology, understanding how AI can process and generate information across various data types – text, images, audio, and video – will unlock unprecedented capabilities for research, development, and patient care. Traditional AI models often specialize in one modality, like processing text for drug discovery or analyzing images for diagnostics. Multimodal AI, however, aims to mimic human cognitive abilities by integrating information from multiple senses, leading to a more holistic understanding and generation of complex data. Consider the complexity of a clinical trial report. It contains structured text, perhaps MRI images, and potentially audio recordings of patient interviews. A unimodal text model might extract drug efficacy data, while an image model identifies abnormalities. A multimodal model, however, could correlate textual descriptions of symptoms with visual evidence from scans and patient vocal inflections, providing a richer, more nuanced analysis. This integrated approach allows for cross-modal reasoning, where insights from one modality can enhance understanding in another, leading to more robust and accurate outcomes.
Architectures for Multimodal Fusion in Biotech and Pharma
The core challenge in multimodal AI lies in effectively fusing information from disparate data types. Different modalities have unique structures and representations. Text is sequential, images are spatial grids of pixels, and audio is a time-series of waveforms. Architectures for multimodal fusion typically involve three main stages: feature extraction , fusion , and prediction/generation . Feature Extraction: Each modality first undergoes a specialized feature extraction process. For text, this might involve transformer-based encoders like BERT or GPT. For images, convolutional neural networks (CNNs) are standard. For audio, techniques like spectrograms coupled with recurrent neural networks (RNNs) or transformers are common. The goal is to transform raw data into a high-dimensional vector representation (embedding) that captures its essential information. Fusion: This is where the magic happens. After individual features are extracted, they need to be combined. Common fusion strategies include: Early Fusion: Concatenating raw data or low-level features before feeding them into a single model. This is simpler but can be sensitive to synchronization issues and may struggle with disparate data types. Late Fusion: Processing each modality independently to generate separate predictions, then combining these predictions (e.g., averaging probabilities) at the end. This is robust but might miss inter-modal relationships. Intermediate Fusion (Joint Representation): The most prevalent approach in advanced multimodal models. Here, features from different modalities are integrated at an intermediate layer, often using attention mechanisms or specialized fusion layers. This allows the model to learn complex relationships and cross-modal alignments. Transformers with cross-attention layers are particularly effective here, allowing tokens from one modality (e.g., text) to attend to tokens from another (e.g., image patches). Prediction/Generation: After fusion, the combined representation is fed into a task-specific head for prediction (e.g., classification, regression) or a decoder for generation (e.g., generating text descriptions from images, creating synthetic audio from text). Let's consider a practical example: drug repurposing. A multimodal AI could take the chemical structure (image/graph data), known biological targets (text data), and gene expression profiles (numerical data) to predict novel therapeutic applications, far more effectively than analyzing each in isolation.
Example: Fusing Text and Image Embeddings for Drug Classification
Imagine we're building a system to classify drugs based on their textual descriptions and their molecular structure diagrams. We'll use pre-trained encoders for each modality and then concatenate their embeddings. import torch from transformers import AutoTokenizer, AutoModel from PIL import Image from torchvision import transforms import io # Assume pre-trained models for text and image # In a real scenario, these would be loaded from Hugging Face or similar class TextEncoder(torch.nn.Module): def __init__(self): super().__init__() self.tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased") self.model = AutoModel.from_pretrained("bert-base-uncased") def forward(self, text): inputs = self.tokenizer(text, return_tensors="pt", padding=True, truncation=True, max_length=512) outputs = self.model(**inputs) return outputs.last_hidden_state[:, 0, :] # [CLS] token embedding class ImageEncoder(torch.nn.Module): def __init__(self): super().__init__() self.preprocess = transforms.Compose([ transforms.Resize(256), transforms.CenterCrop(224), transforms.ToTensor(), transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]), ]) # Using a simplified mock vision model for demonstration # In reality, this would be a ResNet, ViT, etc. self.model = torch.nn.Linear(224*224*3, 768) # Mock: flattens and projects def forward(self, image_data): # image_data is expected to be a PIL Image processed_image = self.preprocess(image_data).unsqueeze(0) # Add batch dim # Mocking a feature extraction: flatten and project return self.model(processed_image.view(processed_image.size(0), -1)) # Initialize encoders text_encoder = TextEncoder() image_encoder = ImageEncoder() # Sample data drug_description = "A selective serotonin reuptake inhibitor (SSRI) commonly used to treat depression." # Mock image data (replace with actual image loading for molecular structure) # For demonstration, let's create a dummy image dummy_image = Image.new('RGB', (224, 224), color = 'red') # Get embeddings text_embedding = text_encoder(drug_description) image_embedding = image_encoder(dummy_image) # Concatenate embeddings (Intermediate Fusion) fused_embedding = torch.cat((text_embedding, image_embedding), dim=1) print(f"Text embedding shape: {text_embedding.shape}") # e.g., torch.Size([1, 768]) print(f"Image embedding shape: {image_embedding.shape}") # e.g., torch.Size([1, 768]) print(f"Fused embedding shape: {fused_embedding.shape}") # e.g., torch.Size([1, 1536]) # This fused_embedding can now be fed into a downstream classifier # e.g., torch.nn.Linear(1536, num_classes) The fused embedding now contains information from both modalities. A subsequent classification layer can learn to make predictions based on this richer, combined representation. This is a common pattern for tasks like drug-target interaction prediction or adverse event detection where both textual reports and visual evidence (e.g., patient photos, medical scans) are relevant.
Example: Generating Text from Multimodal Inputs (e.g., Patient Report Generation)
Another powerful application is generating text based on multimodal inputs. Imagine generating a summary report from a patient's medical scans and transcribed audio notes. This requires a multimodal encoder and a text decoder. import torch from transformers import VisionEncoderDecoderModel, AutoTokenizer, AutoFeatureExtractor from PIL import Image # For demonstration, we'll use a pre-trained Vision-Encoder-Decoder model # This model is typically trained for image captioning, but the principle applies # to other multimodal generation tasks. # In a real biotech/pharma scenario, you might fine-tune such a model # on domain-specific medical images and text. # Load pre-trained model and tokenizer/feature extractor model_name = "nlpconnect/vit-gpt2-image-captioning" model = VisionEncoderDecoderModel.from_pretrained(model_name) tokenizer = AutoTokenizer.from_pretrained(model_name) feature_extractor = AutoFeatureExtractor.from_pretrained(model_name) # Set model to evaluation mode model.eval() # Example: Generate a caption for a medical image # For simplicity, let's use a dummy image. In practice, this would be a CT scan, X-ray, etc. dummy_medical_image = Image.new('RGB', (500, 500), color = 'blue') # Represents a scan # Preprocess the image pixel_values = feature_extractor(images=dummy_medical_image, return_tensors="pt").pixel_values # Generate text (e.g., a diagnostic report snippet) # You can customize generation parameters like max_length, num_beams, etc. with torch.no_grad(): generated_ids = model.generate(pixel_values, max_length=50, num_beams=4, early_stopping=True) generated_text = tokenizer.decode(generated_ids[0], skip_special_tokens=True) print(f"Generated text from medical image: \"{generated_text}\"") # While this example uses image-to-text, the concept extends to fusing text/audio # with the image encoder output before feeding to the decoder. # For instance, you could concatenate audio embeddings with image embeddings # and then use a modified decoder that takes this combined input. This snippet demonstrates the generative aspect. While the example uses a standard image captioning model, the underlying principle of an encoder-decoder architecture is key. For more complex multimodal inputs (e.g., image + audio + text description), the encoder part would be extended to process and fuse all these modalities into a unified representation, which the decoder then uses to generate text.
Key Takeaways for Pharmacy and Biotech:
Enhanced Diagnostics: Combine imaging (MRI, CT, histology) with patient reports (text) and audio recordings (e.g., heart sounds, patient interviews) for more accurate disease detection and prognosis. Accelerated Drug Discovery: Integrate chemical structures (image/graph), genomic data (text/numerical), and scientific literature (text) to predict drug efficacy, adverse effects, and repurposing opportunities. Personalized Medicine: Analyze patient-specific data across modalities (genetics, medical history, lifestyle) to tailor treatment plans and predict individual responses. Automated Report Generation: Generate comprehensive patient summaries or research reports from diverse raw data streams, saving time and improving consistency. Clinical Trial Optimization: Use multimodal data to identify suitable patient cohorts, monitor treatment responses more comprehensively, and predict trial outcomes.
Practice Exercise: Multimodal Application Brainstorm
You are tasked with designing a novel generative AI application for a pharmaceutical company. This application must leverage at least three different modalities (e.g., text, image, audio, video, structured data). Describe your application idea, detailing: The problem it aims to solve in pharmacy or biotechnology. The specific modalities it would ingest as input. How these modalities would be processed and fused (e.g., early, late, or intermediate fusion, specific model types). The type of output it would generate (e.g., text, image, numerical prediction, or a combination). The potential impact or benefit of this multimodal approach compared to a unimodal solution. Think creatively about how combining different types of information can lead to insights or capabilities currently difficult to achieve.</
Watch the full lesson — free
This topic is part of Build & Ship Generative AI Applications, a complete AI-narrated video course. Press play once and watch the entire lecture like a movie.
Start the course free →