Lesson · 40 min · Free
Text Representation
Text Representation body { font-family: sans-serif; line-height: 1.6; margin: 20px; } h1, h2 { color: #2c3e50; } p { margin-bottom: 1em; } pre { background-color: #ecf0f1; padding: 15px; border-radius: 5px; overflow-x: a
Text Representation
In the burgeoning field of AI in Drug Discovery, much of the foundational data exists not as structured numerical tables, but as unstructured text. This includes scientific literature, patent descriptions, clinical trial reports, patient records, and even chemical compound names (e.g., IUPAC nomenclature). To harness the power of machine learning and deep learning algorithms, this textual information must be converted into a numerical format that computers can understand and process. This process is known as text representation or text embedding . Effective text representation is crucial for tasks such as identifying novel drug targets, predicting drug-target interactions, extracting adverse drug reaction information, or automating literature reviews. Without robust numerical representations, the semantic meaning and contextual relationships within the text remain inaccessible to AI models.
Methods of Text Representation
Historically, text representation began with relatively simple statistical methods, evolving into sophisticated neural network-based approaches. Each method attempts to capture different aspects of language, from word frequency to complex semantic relationships.
Bag-of-Words (BoW)
One of the earliest and simplest methods, Bag-of-Words (BoW), represents a document as an unordered collection of words, disregarding grammar and word order. It essentially counts the frequency of each word in a document. While straightforward, BoW can suffer from high dimensionality (many unique words) and loses crucial contextual information. from sklearn.feature_extraction.text import CountVectorizer documents = [ "Aspirin is a non-steroidal anti-inflammatory drug.", "Inflammation is a key process in many diseases.", "Drug discovery often involves screening many compounds." ] # Create a CountVectorizer object vectorizer = CountVectorizer() # Fit and transform the documents bow_matrix = vectorizer.fit_transform(documents) # Print the vocabulary learned print("Vocabulary:", vectorizer.get_feature_names_out()) # Print the BoW matrix print("BoW Matrix:\n", bow_matrix.toarray()) In the output, each row corresponds to a document, and each column corresponds to a unique word in the entire corpus, with values indicating word counts.
TF-IDF (Term Frequency-Inverse Document Frequency)
TF-IDF is an improvement over BoW, addressing the issue where common words (like "the", "is", "a") might dominate the representation simply due to their high frequency. TF-IDF assigns a weight to each word that reflects its importance in a document relative to the entire corpus. High TF-IDF scores are given to words that appear frequently in a specific document but infrequently across other documents. from sklearn.feature_extraction.text import TfidfVectorizer documents = [ "Aspirin is a non-steroidal anti-inflammatory drug.", "Inflammation is a key process in many diseases.", "Drug discovery often involves screening many compounds." ] # Create a TfidfVectorizer object vectorizer = TfidfVectorizer() # Fit and transform the documents tfidf_matrix = vectorizer.fit_transform(documents) # Print the vocabulary learned print("Vocabulary:", vectorizer.get_feature_names_out()) # Print the TF-IDF matrix print("TF-IDF Matrix:\n", tfidf_matrix.toarray()) The TF-IDF matrix provides a more nuanced numerical representation, where words more specific to a document have higher weights.
Word Embeddings (Word2Vec, GloVe, FastText)
Modern approaches leverage neural networks to create "word embeddings," which are dense vector representations where words with similar meanings are located closer to each other in a multi-dimensional space. These models learn context by analyzing words in their surrounding text. Word2Vec (developed by Google) is a prominent example, offering two architectures: Skip-gram and Continuous Bag-of-Words (CBOW). Skip-gram: Predicts surrounding words given a central word. CBOW: Predicts a central word given its surrounding context words. Other popular word embedding models include GloVe (Global Vectors for Word Representation) and FastText (which considers sub-word information).
Contextual Embeddings (BERT, GPT, etc.)
The latest advancements involve "contextual embeddings" from large pre-trained language models like BERT (Bidirectional Encoder Representations from Transformers), GPT (Generative Pre-trained Transformer), and their successors. Unlike traditional word embeddings where a word has a single fixed vector, contextual embeddings generate a vector for a word based on its specific context in a sentence. This allows models to disambiguate word meanings (e.g., "bank" as a financial institution vs. a river bank) and capture complex semantic relationships. These models are often pre-trained on vast amounts of text data and then fine-tuned for specific downstream tasks in drug discovery, offering state-of-the-art performance in natural language processing (NLP) applications.
SMILES and Beyond
While the above methods focus on natural language, chemical structures themselves can also be represented as text strings, most notably using SMILES (Simplified Molecular-Input Line-Entry System). SMILES strings can then be processed using NLP techniques or specialized deep learning architectures (e.g., recurrent neural networks, transformers) to learn representations of molecules. This bridges the gap between chemical information and text-based AI models. Key Takeaway 1: Text representation transforms unstructured textual data into numerical vectors understandable by AI algorithms. Key Takeaway 2: Methods range from simple frequency-based (BoW, TF-IDF) to sophisticated neural network-based embeddings (Word2Vec, BERT). Key Takeaway 3: Word embeddings capture semantic relationships, with contextual embeddings offering superior understanding of polysemy and nuance. Key Takeaway 4: The choice of representation method depends on the specific task, available data, and computational resources. Key Takeaway 5: Chemical structures can also be represented as text (e.g., SMILES) for AI processing.
Practice Exercise
Consider a scenario where you are building an AI system to identify potential adverse drug reactions (ADRs) from a corpus of clinical notes. These notes contain free-form text describing patient symptoms, administered drugs, and outcomes. Briefly explain why a simple Bag-of-Words (BoW) approach might be insufficient for this task and suggest at least one more advanced text representation method that would be more suitable, justifying your choice. Think about the specific challenges of identifying ADRs from clinical text.
Watch the full lesson — free
This topic is part of AI in Drug Discovery, a complete AI-narrated video course. Press play once and watch the entire lecture like a movie.
Start the course free →