Lesson · 40 min · Free
Command-Line Data Pipelines
Command-Line Data Pipelines body { font-family: sans-serif; line-height: 1.6; margin: 20px; } h1, h2 { color: #2c3e50; } pre { background-color: #ecf0f1; padding: 15px; border-radius: 5px; overflow-x: auto; } code { font
Command-Line Data Pipelines
In bioinformatics and computational genomics, we often deal with large datasets that require a series of sequential processing steps. Manually executing each step can be tedious, error-prone, and inefficient. This is where command-line data pipelines become indispensable. A pipeline is essentially a series of commands chained together, where the output of one command serves as the input for the next. This approach offers reproducibility, scalability, and efficiency, crucial for robust scientific research. The power of the command line lies in its ability to compose complex operations from simple, atomic tools. Unix-like operating systems (Linux, macOS) provide a rich ecosystem of utilities that can be combined using redirection and pipes. Redirection allows us to send the output of a command to a file ( > for overwrite, >> for append) or take input from a file ( < ). Pipes ( | ) are even more powerful, directing the standard output (stdout) of one command to the standard input (stdin) of another, effectively linking them together. Consider a common scenario: you have raw sequencing reads in a FASTQ file, and you want to count the occurrences of a specific adapter sequence. You might first want to filter out low-quality reads, then search for the adapter, and finally count its occurrences. Each of these steps can be handled by a distinct command-line tool, and a pipeline allows us to string them together seamlessly.
Building Basic Pipelines with Pipes and Redirection
Let's illustrate with a simple example. Suppose we have a file named sample_data.txt containing a list of gene names, some of which are duplicates, and we want to get a unique, sorted list of these genes. # Create a sample file for demonstration echo -e "geneA\ngeneB\ngeneC\ngeneA\ngeneD\ngeneB" > sample_data.txt # Pipeline to sort and find unique lines cat sample_data.txt | sort | uniq In this pipeline: cat sample_data.txt : This command reads the content of sample_data.txt and prints it to standard output. | (pipe): The output of cat is piped as input to the sort command. sort : This command sorts the lines alphabetically and prints the sorted output to standard output. | (pipe): The sorted output is then piped as input to the uniq command. uniq : This command filters out adjacent duplicate lines, printing only unique lines to standard output. The final output would be: geneA geneB geneC geneD This simple example demonstrates the fundamental concept of chaining commands. In bioinformatics, these commands would be specialized tools like fastqc , trim_galore , bwa mem , samtools , vcftools , bedtools , etc. The ability to combine them flexibly is what makes command-line pipelines so powerful. Let's consider a slightly more complex bioinformatics-relevant example. Imagine you have a FASTQ file ( reads.fastq ) and you want to count how many reads contain the sequence "ADAPTER_SEQ". You might use grep for pattern matching and wc -l for line counting. # Assuming reads.fastq exists and contains sequencing reads # Example: Create a dummy reads.fastq for demonstration echo -e "@read1\nAGCTAGCTADAPTER_SEQAGCT\n+\nIIIIIIIIIIIIIIIIIIIIII\n@read2\nTGATCGATCGATCGATCG\n+\nIIIIIIIIIIIIIIIIII\n@read3\nADAPTER_SEQADAPTER_SEQ\n+\nIIIIIIIIIIIIIIIIIIIIII\n" > reads.fastq # Pipeline to count reads containing "ADAPTER_SEQ" grep "ADAPTER_SEQ" reads.fastq | wc -l In this pipeline: grep "ADAPTER_SEQ" reads.fastq : This command searches for lines containing "ADAPTER_SEQ" within reads.fastq and prints only those matching lines to standard output. | (pipe): The output (matching lines) from grep is piped as input to wc -l . wc -l : This command counts the number of lines it receives from standard input and prints the count. The result would be 2 , as two reads contain the specified adapter sequence. For more complex pipelines, you might save intermediate results to files using redirection ( > ) for debugging or for steps that are very resource-intensive and shouldn't be re-run unnecessarily. However, for many common operations, piping directly between commands is more efficient as it avoids writing temporary files to disk.
Key Takeaways
Command-line pipelines chain multiple commands together to perform complex data processing tasks. Pipes ( | ) direct the standard output of one command to the standard input of another. Redirection ( > , >> , < ) allows managing input and output with files. Pipelines promote reproducibility, efficiency, and scalability in bioinformatics workflows. Understanding common Unix utilities ( cat , sort , uniq , grep , wc ) is fundamental.
Practice Exercise
You have a FASTA file named sequences.fasta . Write a single command-line pipeline that will: Extract all sequence headers (lines starting with > ). Sort these headers alphabetically. Count the total number of unique headers. Assume sequences.fasta contains: >geneA_species1 ATGCGTACGT >geneB_species2 CGTAGCTA >geneA_species1 GCTAGCTA >geneC_species1 ATCGATCG What would be the final output of your pipeline?
Watch the full lesson — free
This topic is part of Bioinformatics & Computational Genomics, a complete AI-narrated video course. Press play once and watch the entire lecture like a movie.
Start the course free →