Text Representation in AI

Text Representation in AI

Neural networks operate on numbers, not words. Before any deep learning model can process a clinical note, a research abstract, or a drug interaction report, the raw text must be converted into a numerical representation. This process is called text representation or text vectorization , and it is one of the most consequential design decisions in any natural language processing (NLP) pipeline. The quality of the representation directly limits what the model can learn downstream. The simplest approach is one-hot encoding , where each word in the vocabulary is assigned an index and represented as a sparse binary vector. A sentence of medical terms like "patient responded well to treatment" becomes a sequence of vectors, each with a single 1 and thousands of zeros. One-hot encoding is intuitive but has serious drawbacks: the vectors are extremely high-dimensional, they are sparse, and they carry no notion of similarity. The vectors for "aspirin" and "ibuprofen" are just as different from each other as either is from "hospital" , even though the first two are semantically related drugs. A more powerful family of methods addresses this with dense embeddings . Techniques such as Word2Vec, GloVe, and fastText learn vectors of modest size (typically 100–300 dimensions) in which semantically similar words end up close together in the vector space. In a well-trained embedding space, the vectors for "acetaminophen" and "paracetamol" — two names for the same molecule — will be nearly identical, and arithmetic relationships can even emerge, such as king − man + woman ≈ queen . For biomedical applications, embeddings trained on corpora like PubMed abstracts or clinical notes (for example, BioWordVec) capture domain-specific relationships such as drug–target or disease–gene proximity. In modern practice, embeddings are usually learned jointly with the task inside the neural network itself. In PyTorch, the nn.Embedding layer is a learnable lookup table: each row corresponds to one word in the vocabulary, and during training the gradient updates adjust these rows so that the representation becomes useful for the prediction at hand. This means a model trained to classify adverse drug event reports will, over time, develop its own internal representation of drug names and clinical terms — one tailored to that specific task. Finally, keep in mind that text is a sequence . A sentence is not a bag of independent words; order matters ( "morphine given before surgery" vs. "surgery given before morphine" ). Representations therefore typically preserve sequence structure by producing one vector per token, which is then fed to sequence-aware architectures such as recurrent networks, LSTMs, or Transformers. The pipeline almost always follows the same pattern: tokenize → map tokens to integer IDs → look up (or learn) embeddings → feed the sequence of vectors to the model . The first example below builds a minimal vocabulary and converts a short clinical sentence into a sequence of integer token IDs — the step that always precedes embedding lookup. import torch # A tiny corpus of clinical sentences corpus = [ "patient responded well to treatment", "treatment reduced patient symptoms", "drug caused mild side effects" ] # Build a vocabulary: map each unique word to an integer ID tokens = set() for sentence in corpus: tokens.update(sentence.split()) word_to_id = {word: i for i, word in enumerate(sorted(tokens))} print("Vocabulary size:", len(word_to_id)) print("Vocabulary:", word_to_id) # Convert a sentence into a tensor of token IDs def encode(sentence, word_to_id): ids = [word_to_id[word] for word in sentence.split()] return torch.tensor(ids, dtype=torch.long) sentence = "treatment reduced patient symptoms" token_ids = encode(sentence, word_to_id) print("Token IDs:", token_ids) print("Shape:", token_ids.shape) The second example takes those integer IDs and passes them through a learnable nn.Embedding layer, converting each token ID into a dense vector. Note that the embedding layer is part of the model and its weights are updated by backpropagation during training. import torch import torch.nn as nn vocab_size = 20 # number of unique words in the vocabulary embedding_dim = 16 # length of each dense word vector # Learnable lookup table: maps token IDs to dense vectors embedding_layer = nn.Embedding(num_embeddings=vocab_size, embedding_dim=embedding_dim) # Example token IDs for the sentence "treatment reduced patient symptoms" token_ids = torch.tensor([14, 5, 9, 17]) # Look up the dense vectors for each token embedded = embedding_layer(token_ids) print("Embedded shape:", embedded.shape) # (4 tokens, 16 dims) print("First word vector:", embedded[0]) # Embeddings are trainable parameters print("Trainable parameters:", embedding_layer.weight.numel()) Neural networks require numerical input, so text must be converted into vectors before modeling. One-hot encoding is simple but sparse, high-dimensional, and captures no semantic similarity between words. Dense embeddings (Word2Vec, GloVe, BioWordVec) place semantically related terms close together in vector space. nn.Embedding in PyTorch is a learnable lookup table whose vectors are optimized during training for the specific task. Text is sequential: representations preserve one vector per token so that order-sensitive architectures (RNNs, LSTMs, Transformers) can use it. The standard pipeline is tokenize → map to integer IDs → embed → feed to the model. Practice exercise: Using the vocabulary-building code from the first example, add two new clinical sentences to the corpus (for example, about a drug and a dosage), rebuild the vocabulary, and encode one of the new sentences into token IDs. Then create an nn.Embedding layer with embedding_dim=32 , pass your token IDs through it, and verify that the output shape is (number_of_tokens, 32) . Finally, write a short paragraph explaining why two different drug names would receive very different one-hot vectors but might receive similar learned embeddings after training on a task such as adverse event classification.