Pipelines of Text: Combining Commands and Moving Data

Pipelines of Text: Combining Commands and Moving Data

In computational biomedicine, most of the data you will encounter—gene expression matrices, FASTQ sequencing reads, variant call files, clinical records—exists as plain text. The Unix shell was designed around a simple but powerful idea: programs should do one thing well, and complex tasks should be accomplished by chaining small tools together. This is the pipeline philosophy , and mastering it is the difference between manually editing thousands of files and automating the entire workflow in a single line of code. The key mechanism is the pipe operator ( | ), which connects the standard output ( stdout ) of one command directly to the standard input ( stdin ) of the next. Instead of writing intermediate results to temporary files and then processing them step by step, the pipe streams data through a series of filters in real time. For example, imagine you have a gene expression file with thousands of rows and you want to find the top expressed genes in a cohort. Rather than opening the file in an editor, you can sort, filter, and count in one continuous flow. Each stage of the pipeline is a small, testable unit—if a result looks wrong, you can inspect the output of any intermediate command by temporarily breaking the pipe and examining what flows through. Several commands appear constantly in bioinformatics pipelines. grep searches text using patterns (often regular expressions), sort orders lines, uniq collapses or counts adjacent duplicate lines, cut extracts specific columns from delimited files, wc counts lines, words, or characters, and head and tail view the beginning or end of a stream. Redirection operators complement piping: > writes output to a file, >> appends to a file, and < feeds a file into a command as input. Together, these tools let you build workflows such as extracting all missense variants from a VCF, counting reads per gene, or summarizing patient cohort metadata—tasks that would otherwise require writing custom scripts. Consider a common scenario in pharmacogenomics research: you have a tab-separated file of drug response measurements and want to know how many unique patients show a response above a threshold in a particular treatment arm. The first example below demonstrates a classic pipeline combining column extraction, filtering, sorting, and counting: # Count unique patients with strong response to drug X (column 3 = response score) # File format: patient_id treatment response_score cut -f1,3 drug_response.tsv | awk '$3 > 8.0 {print $1}' | sort -u | wc -l # Step-by-step breakdown: # cut -f1,3 drug_response.tsv → keep patient ID and response columns # awk '$3 > 8.0 {print $1}' → filter rows where response exceeds 8.0 # sort -u → remove duplicate patient IDs # wc -l → count the remaining unique patients The second example illustrates a workflow you will use constantly in transcriptomics: inspecting and summarizing a FASTQ sequencing file. Here, the pipeline extracts sequence quality information and computes a quick summary statistic without ever loading the full file into memory—a critical advantage when working with datasets containing billions of reads: # Preview the first sequence record in a FASTQ file head -n 4 sample_R1.fastq # Count total sequencing reads (a read = 4 lines: header, sequence, +, quality) echo "Total reads:" wc -l < sample_R1.fastq | awk '{print $1 / 4}' # Find all reads containing a specific adapter sequence (e.g., Illumina adapter) grep -c "AGATCGGAAGAG" sample_R1.fastq # Save the list of reads with adapter contamination to a report file grep -B1 "AGATCGGAAGAG" sample_R1.fastq | grep "^@" > adapter_contaminated_reads.txt Notice two important conventions in these examples. First, wc -l < file uses input redirection so that wc reports only the count, not the filename. Second, the final example combines grep -B1 (which prints the line before each match—here, the FASTQ header) with a second grep that keeps only header lines, then redirects the result into a permanent file. This pattern of filter, refine, save mirrors exactly how production bioinformatics pipelines are constructed, from quality control with FastQC to variant calling with GATK. The pipe operator | streams the output of one command directly into the input of the next, enabling complex analysis without intermediate files. Redirection operators ( > , >> , < ) move data between the command line and files, allowing you to save results and feed inputs. Core text-manipulation commands— grep , sort , uniq , cut , wc , head , tail , and awk —are the building blocks of nearly every bioinformatics workflow. Pipelines are composable and debuggable : test each stage independently by breaking the pipe and inspecting intermediate output. Streaming text through pipelines is memory-efficient, making it suitable for large genomic datasets that cannot fit in RAM. In single-cell genomics, the same principles scale up: command-line pipelines preprocess raw FASTQ files before they ever reach tools like Cell Ranger or Seurat. Practice exercise: Download the provided file patient_metadata.tsv , which contains five tab-separated columns: patient ID, age, sex, diagnosis, and treatment arm. Using a single pipeline, answer the following question: How many female patients over age 50 are enrolled in the "combination_therapy" arm? Build your solution incrementally—first use cut to view the relevant columns, then add awk to apply the age and sex filters, then extend the pipeline to count unique patient IDs. Once your count is correct, redirect the full list of matching patient IDs into a file called eligible_patients.txt and verify its contents with cat . Compare your one-line solution with a neighbor: there are often several valid pipeline orderings, and discussing the trade-offs (readability versus brevity) is excellent preparation for writing your own multi-stage genomics workflows.