Lesson · 40 min · Free
LLM Math Foundations
LLM Math Foundations body { font-family: sans-serif; line-height: 1.6; margin: 20px; } h1, h2 { color: #2c3e50; } pre { background-color: #ecf0f1; padding: 15px; border-radius: 5px; overflow-x: auto; } code { font-family
LLM Math Foundations
Welcome to the "LLM Math Foundations" lesson, a crucial stepping stone in understanding how Large Language Models (LLMs) operate, particularly in the context of drug discovery. While LLMs primarily deal with text, their underlying mechanisms are deeply rooted in mathematics. For pharmacy and biotech students, grasping these foundational concepts is essential for critically evaluating LLM outputs, designing effective prompts, and even contributing to the development of new AI tools in your field. This lesson will focus on the core mathematical principles that enable LLMs to process, understand, and generate human-like text. At its heart, an LLM treats language as a sequence of numerical representations. Each word, or more accurately, each "token" (which can be a word, part of a word, or punctuation), is converted into a high-dimensional vector. These vectors are not arbitrary; they are learned representations that capture the semantic meaning and contextual relationships of the tokens. Words with similar meanings or contexts will have vector representations that are "close" to each other in this high-dimensional space. This process, known as "word embedding" or "token embedding," is fundamental to how LLMs interpret input.
Vector Operations and Similarity
Once tokens are represented as vectors, LLMs perform various mathematical operations on these vectors. A key operation is calculating the similarity between two vectors. This is often done using the cosine similarity metric, which measures the cosine of the angle between two vectors. A cosine similarity close to 1 indicates high similarity (vectors pointing in roughly the same direction), while a value close to -1 indicates dissimilarity (vectors pointing in opposite directions), and 0 indicates orthogonality (no linear relationship). This metric is vital for tasks like identifying related concepts, retrieving information, and ensuring the coherence of generated text. Another critical aspect is matrix multiplication, which forms the backbone of transformer architectures – the dominant architecture for modern LLMs. Attention mechanisms, which allow LLMs to weigh the importance of different parts of the input sequence when processing a particular token, heavily rely on matrix multiplications to compute query, key, and value vectors, and subsequently, attention scores. These scores are then used to create a weighted sum of value vectors, effectively focusing the model's "attention" on relevant information. Consider a simplified example of vector representation for drug names. Let's say we have two drugs, 'Aspirin' and 'Paracetamol', and their simplified vector representations based on their therapeutic class and side effects. In a real LLM, these vectors would be hundreds or thousands of dimensions long, but for illustration: import numpy as np # Simplified vector representations for 'Aspirin' and 'Paracetamol' # (e.g., dimension 0: anti-inflammatory, dimension 1: analgesic, dimension 2: GI side effects) aspirin_vector = np.array([0.9, 0.7, 0.6]) paracetamol_vector = np.array([0.1, 0.9, 0.2]) # Calculate cosine similarity dot_product = np.dot(aspirin_vector, paracetamol_vector) norm_aspirin = np.linalg.norm(aspirin_vector) norm_paracetamol = np.linalg.norm(paracetamol_vector) cosine_similarity = dot_product / (norm_aspirin * norm_paracetamol) print(f"Aspirin Vector: {aspirin_vector}") print(f"Paracetamol Vector: {paracetamol_vector}") print(f"Cosine Similarity: {cosine_similarity:.4f}") # Output will show a relatively low similarity, reflecting different primary mechanisms/side effects Furthermore, the entire learning process of an LLM, from initial training to fine-tuning, relies on optimization algorithms, predominantly gradient descent and its variants (e.g., Adam). These algorithms adjust the vast number of parameters (weights and biases) within the neural network to minimize a "loss function." The loss function quantifies how far the model's predictions are from the actual desired outputs. Calculus, specifically derivatives, is used to determine the direction and magnitude of these parameter adjustments, allowing the model to gradually improve its performance over millions of iterations. The output generation in LLMs often involves probability distributions. When an LLM predicts the next word in a sequence, it doesn't just pick one word; it calculates a probability distribution over its entire vocabulary. For example, after "The patient experienced severe...", the model might assign high probabilities to "pain," "nausea," or "dizziness." Techniques like softmax are used to convert raw model outputs (logits) into these interpretable probabilities. Sampling methods (e.g., greedy sampling, top-k sampling, nucleus sampling) are then applied to select the actual word to generate, balancing coherence with creativity. import torch import torch.nn.functional as F # Example logits from an LLM for predicting the next word # (e.g., 'pain', 'nausea', 'dizziness', 'recovery', 'synergy') # In a real scenario, this would be a much larger tensor for the entire vocabulary logits = torch.tensor([2.5, 1.8, 2.2, 0.5, -1.0]) # Apply softmax to convert logits into probabilities probabilities = F.softmax(logits, dim=0) # Get the index of the highest probability word (greedy approach) predicted_index = torch.argmax(probabilities) # Assuming a vocabulary for illustration vocabulary = ['pain', 'nausea', 'dizziness', 'recovery', 'synergy'] predicted_word = vocabulary[predicted_index] print(f"Logits: {logits}") print(f"Probabilities: {probabilities}") print(f"Predicted Word (Greedy): {predicted_word}") Understanding these mathematical underpinnings empowers you to look beyond the "black box" of LLMs. It allows you to appreciate why certain prompts work better than others, how model biases might emerge from the training data's statistical properties, and the inherent probabilistic nature of LLM outputs. For drug discovery, this means being able to critically assess LLM suggestions for novel compounds, interpret generated hypotheses about drug-target interactions, and understand the limitations of such AI tools.
Key Takeaways
LLMs represent text as high-dimensional numerical vectors (embeddings). Vector operations, particularly cosine similarity, are used to measure semantic relationships. Matrix multiplication is fundamental to transformer architectures and attention mechanisms. Optimization algorithms (e.g., gradient descent) adjust model parameters based on a loss function. LLM output generation involves calculating probability distributions over vocabulary, converted via softmax. Understanding these mathematical concepts is crucial for critical evaluation and effective utilization of LLMs in biotech.
Practice Exercise
Imagine an LLM is being used to analyze patient reports for adverse drug reactions (ADRs). If the LLM assigns a high cosine similarity between the vector for "rash" and the vector for "urticaria," what does this mathematically imply about these two terms within the model's understanding? Furthermore, if you were to fine-tune this LLM on a specific dataset of dermatological ADRs, how might the vector representations and their similarities for these terms change, and why?
Watch the full lesson — free
This topic is part of AI in Drug Discovery, a complete AI-narrated video course. Press play once and watch the entire lecture like a movie.
Start the course free →