Lesson · 40 min · Free
Training-Data Preparation & Optimization
Training-Data Preparation & Optimization Generative AI models are only as good as the data used to train or fine-tune them. For pharmaceutical and biotech applications—where models may assist with literature summariz
Training-Data Preparation & Optimization
Generative AI models are only as good as the data used to train or fine-tune them. For pharmaceutical and biotech applications—where models may assist with literature summarization, adverse-event detection, or regulatory writing—careful data preparation is not optional; it directly determines model accuracy, safety, and regulatory defensibility. In this lesson, you will learn how to collect, clean, structure, and optimize datasets for fine-tuning large language models (LLMs). Training data for LLM fine-tuning is typically organized as instruction–response pairs , often stored in JSON Lines (JSONL) format. Each record contains a prompt (the instruction or question) and a completion (the desired output). Domain-specific data—such as drug–drug interaction pairs, clinical trial summaries, or pharmacovigilance reports—must be de-identified, deduplicated, and formatted consistently before training. Poorly prepared data leads to hallucinations, biased outputs, and wasted compute resources. Data cleaning involves removing duplicates, correcting formatting inconsistencies, filtering out protected health information (PHI), and normalizing terminology (e.g., standardizing drug names to a controlled vocabulary such as RxNorm). Data optimization goes further: balancing the dataset across categories, curating high-quality examples over sheer volume, and splitting data into training, validation, and test sets. Research consistently shows that a few thousand high-quality, well-curated examples often outperform millions of noisy ones.
Example 1: Preparing a JSONL Fine-Tuning File
The following Python script converts a CSV of pharmacy Q&A records into the JSONL format required by most fine-tuning APIs, while performing basic cleaning and de-identification: import pandas as pd import json import re # Load raw data from a CSV of pharmacy Q&A pairs df = pd.read_csv("pharmacy_qa_raw.csv") def clean_text(text): text = str(text).strip() # Remove extra whitespace and control characters text = re.sub(r"\s+", " ", text) # Simple de-identification: mask phone numbers and emails text = re.sub(r"\b\d{3}[-.]?\d{3}[-.]?\d{4}\b", "[PHONE]", text) text = re.sub(r"\b[\w.+-]+@[\w-]+\.[\w.]+\b", "[EMAIL]", text) return text records = [] for _, row in df.iterrows(): question = clean_text(row["question"]) answer = clean_text(row["answer"]) # Skip low-quality or empty examples if len(question)
Example 2: Splitting and Validating the Dataset
Before training, always split your data and run validation checks. This script creates a stratified train/validation split and verifies token lengths to avoid exceeding model context limits: import json import random def load_jsonl(path): with open(path) as f: return [json.loads(line) for line in f] def estimate_tokens(text): # Rough approximation: ~4 characters per token return len(text) // 4 data = load_jsonl("pharmacy_finetune.jsonl") random.seed(42) random.shuffle(data) # 90/10 train/validation split split_idx = int(0.9 * len(data)) train_set, val_set = data[:split_idx], data[split_idx:] # Validate token lengths against a 4,096-token context limit MAX_TOKENS = 4096 valid_train = [] for ex in train_set: total_text = " ".join(m["content"] for m in ex["messages"]) if estimate_tokens(total_text)
Key Takeaways
Quality beats quantity: A few thousand clean, diverse, expert-reviewed examples typically outperform large volumes of noisy data. Use JSONL instruction–response format: Most fine-tuning pipelines expect structured message pairs with consistent system prompts. De-identify and comply: Remove PHI and follow HIPAA/GDPR requirements before using any clinical or patient-derived data. Standardize terminology: Normalize drug names and clinical terms using controlled vocabularies (RxNorm, SNOMED CT, MeSH) to improve consistency. Always split your data: Hold out a validation set to detect overfitting and evaluate generalization before deployment. Check token limits: Validate that every example fits within the model's context window to avoid training errors.
Practice Exercise
Download a small public dataset of drug information (e.g., FDA drug labels or a DrugBank sample) and build your own preparation pipeline. Write a Python script that (1) extracts at least 50 question–answer pairs, such as "What is the mechanism of action of metformin?" paired with the corresponding label text, (2) cleans and de-identifies the text, (3) converts the pairs into the JSONL message format shown in Example 1, and (4) performs a train/validation split with token-length validation as in Example 2. Then write a short reflection (150–200 words) identifying two data-quality issues you encountered and how you resolved them—this mirrors the iterative curation process used in real pharmaceutical AI projects.
Watch the full lesson — free
This topic is part of Build & Ship Generative AI Applications, a complete AI-narrated video course. Press play once and watch the entire lecture like a movie.
Start the course free →