Lesson · 40 min · Free
Text Representation Basics
Text Representation Basics body { font-family: Arial, sans-serif; line-height: 1.6; margin: 20px; } h1, h2 { color: #2c3e50; } pre { background-color: #ecf0f1; padding: 10px; border-radius: 5px; overflow-x: auto; } code
Text Representation Basics
In the realm of Artificial Intelligence, particularly within drug discovery, much of the valuable information is locked within unstructured text data. This includes scientific literature, clinical trial reports, patient records, and even chemical patents. To enable AI models to process and understand this text, it must first be converted into a numerical format. This process, known as text representation or text embedding, is fundamental to natural language processing (NLP) applications in bioinformatics and cheminformatics. At its core, text representation aims to capture the semantic meaning and contextual relationships of words, phrases, or entire documents as vectors in a multi-dimensional space. The principle is that words with similar meanings or contexts should be positioned closer to each other in this vector space. This allows AI algorithms, which typically operate on numerical data, to perform tasks like classification, clustering, information retrieval, and even generate new text. Early methods for text representation were often based on simple statistical counts. One of the most straightforward is the Bag-of-Words (BoW) model. In BoW, a text (like a sentence or document) is represented as a bag (multiset) of its words, disregarding grammar and even word order, but keeping multiplicity. Each unique word in the entire corpus forms a dimension in a vector. A document's vector then contains the frequency of each word in that document. While simple, BoW can be effective for tasks where word order is less critical.
From Bags to Embeddings: Capturing Meaning
A common refinement of the Bag-of-Words model is TF-IDF (Term Frequency-Inverse Document Frequency). TF-IDF not only considers how often a word appears in a document (Term Frequency, TF) but also how unique or rare that word is across the entire collection of documents (Inverse Document Frequency, IDF). Words that appear frequently in a specific document but rarely in the corpus are given higher weights, indicating their importance to that document. This helps to filter out common words like "the" or "is" which carry little specific meaning. Let's consider a simple example using TF-IDF. Imagine we have two drug discovery abstracts: "Novel inhibitors of protein kinase C show promise in cancer therapy." "Kinase inhibitors are a key target for cancer treatment." The TF-IDF approach would assign higher weights to terms like "protein kinase C" in the first abstract and "key target" in the second, as they are more specific to their respective contexts compared to common words like "in" or "for". Here's a conceptual Python code snippet illustrating TF-IDF using scikit-learn: from sklearn.feature_extraction.text import TfidfVectorizer documents = [ "Novel inhibitors of protein kinase C show promise in cancer therapy.", "Kinase inhibitors are a key target for cancer treatment." ] # Initialize TF-IDF Vectorizer # max_features can limit the vocabulary size # stop_words='english' removes common English stop words vectorizer = TfidfVectorizer(stop_words='english', max_features=1000) # Fit and transform the documents tfidf_matrix = vectorizer.fit_transform(documents) # Get the feature names (words in the vocabulary) feature_names = vectorizer.get_feature_names_out() print("TF-IDF Matrix Shape:", tfidf_matrix.shape) print("\nFeature Names (Vocabulary):") print(feature_names) print("\nTF-IDF Scores for Document 1:") # Convert sparse matrix to dense array for easier viewing print(tfidf_matrix.toarray()[0]) While TF-IDF provides a robust statistical representation, it still suffers from the "curse of dimensionality" and doesn't inherently capture semantic relationships between words (e.g., "king" and "queen" are related but treated as distinct tokens). Modern approaches often rely on word embeddings, which are dense, low-dimensional vector representations learned from large text corpora. Popular examples include Word2Vec, GloVe, and FastText. These models learn to represent words such that words with similar meanings have similar vector representations. For instance, Word2Vec uses neural networks to predict a word based on its context (Continuous Bag-of-Words, CBOW) or predict the context words given a target word (Skip-gram). The training process results in vectors where semantic relationships are encoded. For example, the vector difference between "king" and "man" might be similar to the difference between "queen" and "woman". Here's a conceptual example of using a pre-trained Word2Vec model (though training one from scratch on a domain-specific corpus would be more appropriate for drug discovery): # This example requires the 'gensim' library and a pre-trained model. # For a real application, you'd load a model trained on biomedical text. # pip install gensim from gensim.models import KeyedVectors # Load a pre-trained Word2Vec model (e.g., Google's News Vectors) # In a real drug discovery scenario, you'd use a model trained on # PubMed abstracts or other relevant biomedical literature. try: model = KeyedVectors.load_word2vec_format( 'GoogleNews-vectors-negative300.bin', binary=True, limit=50000 ) print("Model loaded successfully!") word1 = "drug" word2 = "molecule" word3 = "disease" # Get vector for a word if word1 in model.key_to_index: vec_drug = model[word1] print(f"\nVector for '{word1}' (first 5 dimensions):", vec_drug[:5]) # Find most similar words print(f"\nWords most similar to '{word1}':") if word1 in model.key_to_index: print(model.most_similar(word1, topn=5)) # Calculate similarity between words if word1 in model.key_to_index and word2 in model.key_to_index: similarity = model.similarity(word1, word2) print(f"\nSimilarity between '{word1}' and '{word2}': {similarity:.4f}") # Analogy tasks (e.g., 'king' - 'man' + 'woman' = 'queen') # For drug discovery: 'analgesic' - 'pain' + 'fever' = ? (might give 'antipyretic') if 'analgesic' in model.key_to_index and 'pain' in model.key_to_index and 'fever' in model.key_to_index: result = model.most_similar(positive=['analgesic', 'fever'], negative=['pain'], topn=1) print(f"\n'analgesic' - 'pain' + 'fever' = {result[0][0]}") except FileNotFoundError: print("Pre-trained model 'GoogleNews-vectors-negative300.bin' not found.") print("Please download it (e.g., from Google Drive) or use a biomedical pre-trained model.") print("For demonstration, you can skip this section or train a small model.") except Exception as e: print(f"An error occurred: {e}") These dense vector representations are far more powerful for capturing nuanced meanings and are the foundation for more advanced NLP models like transformers, which have revolutionized AI in recent years. In drug discovery, specialized embeddings trained on vast corpora of biomedical literature (e.g., BioWordVec, ClinicalBERT, PubMedBERT) are crucial for achieving state-of-the-art performance in tasks such as drug-target interaction prediction, adverse drug event detection, and scientific literature mining.
Key Takeaways:
Text data must be converted into numerical representations for AI models to process it. Early methods like Bag-of-Words and TF-IDF rely on word counts and frequencies. TF-IDF weights words based on their importance within a document and across a corpus. Word embeddings (e.g., Word2Vec) learn dense vector representations that capture semantic relationships between words. Specialized embeddings trained on biomedical text are essential for drug discovery applications.
Practice Exercise:
Consider the following two short abstracts from hypothetical drug discovery research: "A novel small molecule inhibitor targeting HER2 was identified, showing promising preclinical activity in breast cancer models." "Clinical trials for a new HER2-targeting antibody drug for metastatic breast cancer are underway, demonstrating good safety profiles." Using the concepts of TF-IDF and word embeddings: Identify at least three words or phrases that would likely receive a high TF-IDF score in abstract 1 compared to a general scientific corpus. Explain why. Discuss how a word embedding model would represent the relationship between "small molecule" and "antibody drug" differently than TF-IDF. What kind of numerical relationship might exist between their vectors?
Watch the full lesson — free
This topic is part of AI in Drug Discovery, a complete AI-narrated video course. Press play once and watch the entire lecture like a movie.
Start the course free →