Automation Superpowers: Wildcards, Variables and Scripts

Automation Superpowers: Wildcards, Variables and Scripts

By this point in the course, you have become comfortable typing individual shell commands to navigate directories, manipulate files, and run bioinformatics tools. But imagine you have just downloaded a dataset of 500 FASTQ files from a sequencing facility. Renaming them one by one, or running a quality-control command 500 separate times, would consume hours of your day and invite repetitive strain injuries and transcription errors. This is where the true power of the command line emerges: automation . Instead of performing tasks manually, you instruct the computer to perform them for you — reliably, reproducibly, and at any scale. For computational biomedicine, where analyses routinely involve thousands of files and multi-step pipelines, automation is not a convenience; it is a professional necessity. The first automation tool is the wildcard , a special character that matches patterns in filenames. The most common wildcard is the asterisk ( * ), which stands for "any number of any characters." If your data directory contains patient_A1.fastq , patient_A2.fastq , and patient_B1.fastq , then the pattern patient_*.fastq matches all three. The shell expands this pattern into a list of matching files before your command even runs, so a single command can act on hundreds of files at once. Wildcards are precise when used carefully: *_R1.fastq selects only the forward reads of paired-end sequencing data, while *.vcf selects all variant call files. This pattern-matching ability, sometimes called "globbing," transforms tedious file management into a one-line operation. The second tool is the variable , a named container for a value you want to reuse. Variables eliminate one of the most dangerous problems in bioinformatics computing: hard-coded paths and parameters scattered throughout your work. If you write DATA_DIR=/home/student/project/rnaseq once at the top of a script, and reference it everywhere as $DATA_DIR , you can move the script to a collaborator's machine or a new project by changing a single line. Variables also make scripts self-documenting — a reader sees REF_GENOME and immediately understands what that path represents. By convention, shell variables are written in uppercase with words separated by underscores, and there must be no spaces around the equals sign, a common source of beginner errors. The third and most powerful tool is the shell script : a plain text file containing a sequence of commands that the shell executes in order. A script turns a fragile sequence of commands you typed from memory into a durable, shareable, re-runnable artifact. In biomedical research, this reproducibility is essential — when your supervisor asks how you processed the data six months ago, or a reviewer questions your pipeline, the script is the answer. A well-structured script begins with a shebang line ( #!/bin/bash ) telling the system which interpreter to use, followed by variable definitions, and then the commands. You run it with bash scriptname.sh , or make it executable with chmod +x . Combined with wildcards and variables, a script of ten lines can replace hundreds of manual commands. Consider a realistic scenario: you have paired-end RNA-seq FASTQ files for multiple patients, and you need to run the quality-control tool fastp on every sample, saving reports into an organized results directory. Doing this manually for each of, say, twenty patients would be slow and error-prone. The following script automates the entire workflow. Study how the variable DATA_DIR centralizes the input location, the for loop combined with the wildcard *_R1.fastq discovers every sample automatically, and string manipulation ( ${sample%_R1.fastq} ) strips the suffix to derive the sample name. #!/bin/bash # run_qc.sh — Run fastp quality control on all paired-end RNA-seq samples DATA_DIR=~/project/rnaseq/raw_data RESULTS_DIR=~/project/rnaseq/qc_results mkdir -p "$RESULTS_DIR" for forward in "$DATA_DIR"/*_R1.fastq; do sample=${forward%_R1.fastq} # strip suffix to get sample name reverse="${sample}_R2.fastq" echo "Processing sample: $(basename "$sample")" fastp -i "$forward" -I "$reverse" \ -o "${sample}_trimmed_R1.fastq" \ -O "${sample}_trimmed_R2.fastq" \ -h "$RESULTS_DIR/$(basename "$sample")_qc.html" \ -j "$RESULTS_DIR/$(basename "$sample")_qc.json" done echo "Quality control complete. Reports saved in $RESULTS_DIR" Notice several professional habits embedded in this script: quoting every variable (protects against filenames containing spaces), using mkdir -p (creates the directory only if it does not exist, without error), and echoing progress messages (so you know the script is running). The backslash at the end of lines simply continues a long command onto the next line for readability. Automation of this kind scales effortlessly: whether you have 20 samples or 20,000, the script is identical. Wildcards and variables also shine in everyday one-line commands, not just full scripts. Suppose you want to count the total number of sequencing reads across all compressed FASTQ files in a directory, and record the result in a summary file. The next example combines a wildcard, a variable, command substitution, and output redirection in a single pipeline — the kind of compact expression you will use daily at the bench-side terminal. # Count total reads across all gzipped FASTQ files and save a summary COUNTS_FILE=~/project/rnaseq/read_counts.txt echo "Read counts per file:" > "$COUNTS_FILE" for f in ~/project/rnaseq/raw_data/*.fastq.gz; do reads=$(zcat "$f" | awk 'END {print NR/4}') echo "$(basename "$f"): $reads reads" >> "$COUNTS_FILE" done echo "---" >> "$COUNTS_FILE" echo "Total files processed: $(ls ~/project/rnaseq/raw_data/*.fastq.gz | wc -l)" >> "$COUNTS_FILE" cat "$COUNTS_FILE" Here, $(...) is command substitution : the shell runs the enclosed command and inserts its output in place. Since each FASTQ record spans four lines, dividing the line count by four yields the number of reads — a small piece of domain knowledge encoded directly into the automation. The > operator creates the summary file and >> appends to it, so results accumulate cleanly. These building blocks — wildcards for selection, variables for reuse, substitution for embedding results, and redirection for output — compose into arbitrarily complex pipelines. Key takeaways from this lesson: Wildcards ( * , ? ) let one command operate on many files by matching filename patterns; the shell expands the pattern before the command runs. Variables store reusable values such as paths and parameters; define them without spaces around = and reference them as $NAME or ${NAME} , always quoted. Shell scripts begin with #!/bin/bash , execute commands in order, and turn manual workflows into reproducible, shareable artifacts. for loops iterate over wildcard-expanded file lists, enabling batch processing of any number of samples with identical code. Command substitution $(...) embeds the output of one command into another, letting scripts compute and use results dynamically. Automation is core scientific practice : scripted pipelines guarantee that you — and your collaborators — can reproduce every processing step exactly. Practice exercise: Create a directory called exercise containing five empty files named sample_01.txt through sample_05.txt (hint: use a wildcard with touch and a for loop). Then write a script named summarize.sh that defines a variable for the directory path, uses a for loop with the wildcard *.txt to visit each file, and appends one line per file — the filename and the current date (hint: date command with command substitution) — to a report file called report.txt . Run your script, inspect the report with cat , and then test its robustness by adding a sixth file and re-running it without modifying the script. If your script processes the new file automatically, you have successfully built your first scalable automation pipeline.