set -euo pipefail in bash

Basics

When asking GenAI to write a bash script, the following line is often added to a script at the top:

set -euo pipefail

This is a shorthand for three separate options that change how bash handles errors. Each option is explained below.

set -e: exit on error

By default, bash continues running even if a command fails. set -e tells bash to exit immediately if any command returns a non-zero exit code. Exit codes are numbers that commands return when they finish: 0 means success, anything else means something went wrong.

set -e

cat file_that_does_not_exist.txt   # this fails
echo "This line should only print if cat succeeded"    # bash exits before reaching this

To follow this behavior generate a file called test_set.sh, add the code from above run it with bash test_set.sh. Try this with and without using the line with set.

With set included should return:

cat: file_that_does_not_exist.txt: No such file or directory

Removing set -e runs:

cat: file_that_does_not_exist.txt: No such file or directory
This line should only print if cat succeeded

set -u: treat unset variables as errors

By default, bash silently substitutes an empty string when a variable is not set. set -u makes bash exit with an error instead.

set -u

echo "$UNDEFINED_VARIABLE"   # exits with: unbound variable

This is particularly relevant when combined with file checks. For example, if a variable holding a filename is accidentally unset, set -u will catch it before the variable is passed to a command.

Running these lines above with test_set.sh should return:

test_set.sh: line 3: UNDEFINED_VARIABLE: unbound variable

while not including set returns an empty line.

set -o pipefail: catch failures in pipes

By default, a pipeline/pipe returns the exit code of the last command only. If in the example below cat fails but sort succeeds, bash records the pipeline as successful (exit code 0) and the failure from cat is invisible to bash. set -o pipefail changes this and the pipeline instead returns the exit code of the first failing command.

In the example below, sort will run in both cases. That is because pipefail does not stop the script. What changes is what bash records as the result of the pipeline, which we made visible by printing $?.

Note: Since pipefail alone does not stop execution that becomes set -e’s job. pipefail’s role is to ensure bash has accurate information that set -e can act upon. Specifically, this ensures that a failure upstream is not silently swallowed before set -e can act on it. Therefore you usually use these two options together with each other.

set -o pipefail

cat file_that_does_not_exist.txt | sort

# Print the error code with `$?` to understand the difference when running with/without set
# echo "Exit code: $?" must immediately follow the pipeline — any command in between resets $?.
echo "Exit code of pipeline: $?"

Adding these lines into test_set.sh should return:

cat: file_that_does_not_exist.txt: No such file or directory
Exit code of pipeline: 1

while not including set returns:

cat: file_that_does_not_exist.txt: No such file or directory
Exit code of pipeline: 0

We see that both times the code runs but returns different exit codes.

Putting it all together

In a normal script we would write something like this:

set -euo pipefail
cat file_that_does_not_exist.txt | sort
echo "Script continued after failed pipeline"

which returns:

cat: file_that_does_not_exist.txt: No such file or directory

Without set we would get:

cat: file_that_does_not_exist.txt: No such file or directory
Script continued after failed pipeline

This section is for students working on an HPC. If you are not yet running jobs on an HPC, you can skip this section.

Slurm job scripts are bash scripts, so the same default behavior applies: without set -euo pipefail, bash will not exit on errors and will silently swallow pipeline failures. However, Slurm adds an important layer: the job’s reported status (COMPLETED vs. FAILED) depends on the final exit code of the script as a whole, not on the exit codes of individual commands within it. This means a job can report COMPLETED even if internal steps failed silently, which can be difficult to detect when running long jobs.

Testing on Crunchomics

We can check the exact behavior on the UvA HPC by creating a file called test_pipefail_slurm.sh:

#!/bin/bash
#SBATCH --job-name=test_pipefail
#SBATCH --output=test_pipefail_%j.out
#SBATCH --error=test_pipefail_%j.err
#SBATCH --time=00:01:00
#SBATCH --ntasks=1

# --- WITHOUT pipefail ---
echo "=== Without pipefail ==="
cat file_that_does_not_exist.txt | sort
echo "Exit code of pipeline: $?"
echo "Script continued"

# --- WITH pipefail and set -e ---
echo "=== With pipefail and set -e ==="
set -euo pipefail
cat file_that_does_not_exist.txt | sort
echo "Exit code of pipeline: $?"
echo "Script continued after pipefail"

Submit this with sbatch test_pipefail_slurm.sh.

This will generate two files:

  • test_pipefail_<JOBID>.out — stdout (what echo prints)
  • test_pipefail_<JOBID>.err — stderr (what cat prints when it fails)

Important: You want to check both files to not miss errors!

The .err file contains (we see that error appears twice because cat fails in both the without and with pipefail sections):

cat: file_that_does_not_exist.txt: No such file or directory
cat: file_that_does_not_exist.txt: No such file or directory

The .out file contains:

=== Without pipefail ===
Exit code of pipeline: 0
Script continued

=== With pipefail and set -e ===

Note that echo “Script continued after pipefail” never prints. That is because set -estopped the script when pipefail exposed the non-zero exit code.

Once the job completes, check what Slurm recorded with sacct -j <JOBID> --format=JobID,JobName,ExitCode,State (replace the jobID with the job number from your script). This should return something like:

JobID           JobName ExitCode      State
------------ ---------- -------- ----------
264615       test_pipe+      1:0     FAILED
264615.batch      batch      1:0     FAILED
264615.exte+     extern      0:0  COMPLETED

Without set -e and pipefail, the same sacct would however return:

JobID           JobName ExitCode      State
------------ ---------- -------- ----------
264524       test_pipe+      0:0  COMPLETED
264524.batch      batch      0:0  COMPLETED
264524.exte+     extern      0:0  COMPLETED

What this demonstrates: without both pipefail and set -e active, a pipeline failure propagates nowhere. It is invisible inside the .out file (the error went to .err), and invisible to Slurm (the job reports COMPLETED). The failure existed, but left no trace that would trigger any alert or downstream check.

When it helps and when it causes problems

These options are standard practice in production scripts and are not wrong. However, for scripts you are actively developing and debugging, they can make troubleshooting harder:

  • pipefail is generally safe and useful to add
  • set -e can cause the script to exit without a clear message, making it hard to know which step failed
  • set -u will exit on unset variables, which is useful but can be surprising if you are not expecting it

Even more important: In computational biology non-zero exit codes are sometimes expected and you might want that the script continues as they don’t always mean something went wrong. A common example is grep: it returns exit code 1 when no match is found, and exit code 2 only when an actual error occurs (e.g. the file does not exist). With set -e active, a grep that finds no match will kill the script, even though “no match found” is a perfectly valid biological result as seen in the example below:

set -euo pipefail

# Create a dummy annotation file for showcasing the command
touch annotations.txt

# This kills the script if the gene is absent from the file
# even though absence is a meaningful biological result 
# since it might tell us that a genome does not encode the gene we are looking for
grep "gene_of_interest" annotations.txt

echo "Pipeline is finished"

grep produces no output when there is no match. The script exits silently with code 1, therefore not printing Pipeline is finished.

Removing the set command returns:

Pipeline is finished

Altogether this means that whether or not we include set -euo pipefail is an important design choice that depends on the tools we are using in our workflow.

Recommendations:

  • While developing and testing a script, leave these options out and add them once the script is working correctly and you understand each step
  • Only add these options if you are sure that every non-zero exit code genuinely represents a failure.
  • If you inherit a script with set -euo pipefail and it exits unexpectedly, this line is often the first place to look.

As an alternative (or complement) to relying on set -e, consider building explicit checks into your workflow using the validation approaches described in the bash input verification tutorial. Checking that an output file exists, is non-empty, and contains the expected number of results after each step is more informative than letting bash decide whether to continue. It also forces you to think about what a correct intermediate result actually looks like, which is a useful habit regardless of whether you use set -euo pipefail or not.

A note on GenAI-generated scripts: GenAI might add set -euo pipefail to code because it is considered best practice for production-level scripts. That is not wrong, but as discussed above this can make the code harder to debug or might not be the behavior we want for our workflow. If a GenAI-generated script adds this to your code, leave it out while developing, verify that every tool in your workflow returns non-zero only on genuine failures, and add it back once the script is working and it is truly needed.