Working with text files

Tip

New to the command line? The Getting started with bash and an HPC tutorial explains these commands step by step. This page is a reference: it lists the key commands with short examples.

Quick reference

Command	What it does	Example
cat	Print a file, or combine files	cat a.txt b.txt > ab.txt
head, tail	Print the first or last lines of a file	head -n 5 a.txt
less	Scroll through a file (q to quit)	less -S a.txt
nano	Edit a file	nano notes.md
wc	Count lines, words or characters	wc -l a.txt
grep	Find lines that match a pattern	grep -c ">" seqs.faa
>, >>	Write the output to a file (>> adds to the end)	ls > files.txt
cut	Extract columns	cut -f1,3 table.txt
sort	Sort lines	sort -k2 table.txt
uniq	Remove or find duplicate lines (sort first)	uniq -d sorted.txt
find	Find files by name, size, date, …	find . -name "*.faa"
sed	Find and replace text	sed 's/old/new/g' a.txt

Wildcards (*, ?, [ ]) and pipes (|) are explained below. For commands that work on files and folders as a whole (copy, move, compress), see Working with files and folders.

Downloading example data

wget -P downloads https://raw.githubusercontent.com/ndombrowski/Unix_tutorial/main/Input_docs/Experiment1.txt
wget -P downloads https://raw.githubusercontent.com/ndombrowski/Unix_tutorial/main/Input_docs/Experiment2.txt
wget -P downloads https://raw.githubusercontent.com/ndombrowski/Unix_tutorial/main/Input_docs/Experiment3.txt

wget -P downloads https://raw.githubusercontent.com/ndombrowski/Unix_tutorial/main/Input_docs/PF00900.faa
wget -P downloads https://raw.githubusercontent.com/ndombrowski/Unix_tutorial/main/Input_docs/PF01015.faa

# Check that the files were downloaded: we expect 5 files
ls downloads/

The content of the files is not important but what you downloaded are simple text files that contain information about a growth experiment and the files look something like this:

Ex1	control1	0.2	0.3	#Ex1_all_good
Ex1	control2	0.3	0.1	#all_good
Ex1	N[0.4uM]	10.1	4.7	#contamination
Ex1	N[0.4uM]	5.2	4.3	#all_good

The faa files are a file type you might encounter when working with molecular data. The FASTA format is a text-based format for representing either nucleotide sequences or amino acid (protein) sequences, in which nucleotides or amino acids are represented using single-letter codes. A sequence begins with a greater-than character (>) followed by a description of the sequence (all in a single line). The lines immediately following the description line are the sequence representation, with one letter per amino acid or nucleic acid, and are typically no more than 80 characters in length.

>GCA_000007185
MVRDKWKDKVWYTILAPDMFDNVEVGETPADDPEKVIGRVLETTLGDVLDDITKHHIKVFFRIYDVEGTTAYSKFEGHRLMRDYVRSLVRRGTSRIDGVIDVVTKDGYKVRVAGLAFTTRRAKTSQQRAIRKEMFKVIEENAKECDFDEFIRRCLSISEEESIPEQIKEAGRKIYPIRQAEIRKTEVLEEPNGLPPYEAVGDRATPELASY
>GCA_000007305
MAAKRATTTRDKWKLKQWYII

In your case you will look at two protein fasta files in which the header contains the information from what genome, i.e. GCA_000007185, a sequence was extracted from as well as the actual sequence itself.

Wildcards: Selecting many files at once

Wildcards (also called globs) are special characters that match file names. The shell replaces the pattern with the list of matching file names before the command runs.

Wildcard	Matches	Example
*	any characters, any number of times (also zero)	downloads/*.txt
?	exactly one character	downloads/Experiment?.txt
[12]	one character: a 1 or a 2	downloads/Experiment[12].txt
[0-9]	one digit	downloads/Experiment[0-9].txt
[A-Z]	one capital letter ([a-z] for a small letter)	downloads/[A-Z]*.faa

# All files that end with .txt
ls downloads/*.txt

# Experiment, followed by exactly one character, followed by .txt
ls downloads/Experiment?.txt

# Only Experiment1.txt and Experiment2.txt
ls downloads/Experiment[12].txt

# Combine wildcards: a capital letter, any characters, a 1 or a 2, and .txt
ls downloads/[A-Z]*[12].txt

Wildcards work with any command that takes file names, e.g. grep -c "control" downloads/*.txt. They only match file names. For patterns inside a file, see Regular expressions.

Viewing files

cat: Printing and combining files

The cat command has three main functions related to manipulating text files:

1. Create a new file using cat. To do this type the command below, then type Hello!, press Enter, and press Ctrl+D to save the file and get the prompt back:

cat > downloads/file_test.txt

2. Display the content of an existing file:

cat downloads/file_test.txt

3. Concatenate, i.e. combine, several files:

# Combine two files into one (here: the same file twice)
cat downloads/file_test.txt downloads/file_test.txt > downloads/file_merged.txt

# Check the content of the new file: Hello! appears twice
cat downloads/file_merged.txt

Exercise

• View the content of downloads/Experiment1.txt by using cat
• Combine downloads/Experiment1.txt and downloads/Experiment2.txt.
• Do the same as above but use a wildcard to not have to type everything and store the output as downloads/Experiments.txt

# Question 1
cat downloads/Experiment1.txt

# Question 2
cat downloads/Experiment1.txt downloads/Experiment2.txt

# Question 3
cat downloads/Experiment*.txt > downloads/Experiments.txt

head and tail: The first and last lines

# Show the first 10 lines
head downloads/PF00900.faa

# Show the first 4 lines (-n sets the number of lines, for head and tail)
head -n 4 downloads/PF00900.faa

# Show the last 10 lines
tail downloads/PF00900.faa

# Show everything from line 2 on, e.g. to skip a header line
tail -n +2 downloads/Experiment1.txt

less: Scrolling through large files

less shows a file one screen at a time. It does not load the whole file, so it also opens very large files quickly.

less downloads/PF00900.faa

# For files with many columns or long lines: do not wrap the lines
less -S downloads/PF00900.faa

• Use the arrow keys or the space bar to move through the file
• With -S, use the left and right arrow keys to see columns further to the right
• Type / followed by a word and press Enter to search for that word
• Type q to exit

nano: Editing files

nano is a basic text editor that runs inside the terminal.

# Open a file (nano makes the file if it does not exist yet)
nano downloads/notes.txt

• Type your text
• Press Ctrl+X to close the file
• Type y to save the changes and press Enter

wc: Counting things

wc (word count) counts the lines, words and characters in a file. It is simple, but essential to check whether your commands did what you expected.

# Count the words in a file
wc -w downloads/Experiment1.txt

# Count the lines in a file
wc -l downloads/Experiment1.txt

grep: Finding patterns in files

grep searches a file for lines that contain a pattern and prints these lines. It is also very useful to check files after you changed them.

# Print all lines that contain the pattern "control"
grep "control" downloads/Experiment1.txt

# Only print the number of matching lines, not the lines themselves
grep -c "control" downloads/Experiment1.txt

# Only find the pattern if it is at the beginning of a line (^)
grep "^Ex" downloads/Experiment1.txt

# Count the sequences in the fasta files: each sequence header starts with ">"
grep -c ">" downloads/*faa

Regular expressions: Patterns inside files

The pattern that grep (and sed) search for inside a file is not a wildcard but a regular expression (regex). Some symbols look the same as wildcards but have a different meaning:

Symbol	As a wildcard (file names)	In a regular expression (file content)
*	any characters, any number of times	the character before it, zero or more times
?	exactly one character	the character before it is optional (with grep -E)
.	a dot	any single character
[12]	a 1 or a 2	a 1 or a 2

Some useful regular expressions:

# Lines with exactly 7 small letters followed by a 1 or 2 (e.g. control1)
grep "[a-z]\{7\}[12]" downloads/Experiment[12].txt

# Lines with 3 to 10 small letters followed by a 1 or 2
grep "[a-z]\{3,10\}[12]" downloads/Experiment[12].txt

By default, grep takes ?, +, { and | literally. With -E (extended regular expressions), grep reads them as special symbols:

# Searches for the literal text "control?", which does not occur in the file
grep "control?" downloads/Experiment1.txt

# With -E, "?" makes the "l" optional: matches "contro" and "control"
grep -E "control?" downloads/Experiment1.txt

# With -E, we can write {7} without the backslashes
grep -E "[a-z]{7}[12]" downloads/Experiment[12].txt

Different programs use slightly different rules for regular expressions. If a pattern does not work as expected, check the manual of the program.

Exercise

• In the faa files, how many sequences do we have? Notice, sequences always start with a >
• In the faa files, how many lines contain 3 consecutive A’s?
• In the faa files, how many lines contain 2 A’s followed by a T?
• In the faa files, how many lines contain 2 M’s followed by a T or a D?

Hint: if you are unsure what is happening, remove the -c to see which lines grep finds.

# Question 1
grep -c ">" downloads/*.faa

# Question 2
grep -c "A\{3\}" downloads/*.faa

# Question 3
grep -c "A\{2\}T" downloads/*.faa

# Question 4
grep -c "M\{2\}[TD]" downloads/*.faa

Redirection: Writing output to a file

By default, a command prints its output to the screen. With > you write the output to a new file instead. With >> you add the output to the end of an existing file.

# Write the list of txt files into a new file
ls downloads/*.txt > downloads/file_list.txt

# Check the content of the new file
cat downloads/file_list.txt

# Add the list of faa files to the end of the same file
ls downloads/*.faa >> downloads/file_list.txt

# Check the content: the faa files are now listed below the txt files
cat downloads/file_list.txt

Warning

> overwrites an existing file without asking. Use >> if you want to keep the existing content.

Pipes (|): Combining commands

A pipe sends the output of one command as input into the next command, instead of to the screen. This lets you combine simple commands into longer ones.

In the example below we first combine two files using cat and use wc -l on the standard output to count the number of lines after concatenating the two files:

cat downloads/Experiment[12].txt | wc -l

Exercise

• Using a pipe, combine the two faa files and count the total number of sequences using grep:

# Question 1
cat downloads/*faa | grep -c ">"

cut: Extracting columns

We can use cut to separate columns. By default cut uses a tab as a delimiter. We could print the first column of our data as follows:

cut -f1 downloads/Experiment1.txt

Or if we only want to get the second column we do this:

cut -f2 downloads/Experiment1.txt

If we wanted to get columns 1 and 2 we do:

cut -f1-2 downloads/Experiment1.txt

We can also change how we cut, i.e. we could cut after a # by using the -d argument:

cut -f1 -d "#" downloads/Experiment1.txt

Exercise

• In PF00900.faa, we want to shorten the fasta header. To do this cut the text of after the first underscore _ (i.e. keep only the first element). Combine this with the head command to only view the first few lines and see if your command worked.
• In PF00900.faa, cut to only keep the text after the first _ (i.e. keep the second element). Again, combine this with the head command.

# Question 1
cut -f1 -d "_" downloads/PF00900.faa | head

# Question 2
cut -f2 -d "_" downloads/PF00900.faa | head

Be careful with the second command: it cuts away the > symbol. Fasta headers must start with >, so after such a command you would need to add the > back in.

sort: Sorting files

sort – sort lines in a file from A-Z and is useful for file organization. We can sort files like this:

sort downloads/Experiment2.txt

Sort by default sorts from left to right, but we also can sort based on a specific column, for example column 4:

sort -k4 downloads/Experiment2.txt

We can also sort by more than one column, i.e. column 5 followed by column 3:

sort -k5 -k3 downloads/Experiment1.txt

uniq: Finding duplicated data

uniq – can be used to remove or find duplicates . However, for this to work the file first needs to be sorted. To do this, we can make use of pipes. Lets, start of by extracting the observations we made with cut:

cut -f5 downloads/Experiment1.txt

We then can sort the column by adding a pipe:

cut -f5 downloads/Experiment1.txt | sort

And only print unique observations with uniq:

cut -f5 downloads/Experiment1.txt | sort | uniq

If we where interested in duplicated observations, we can add the -d argument:

cut -f5 downloads/Experiment1.txt | sort | uniq -d

Check out the manual for more options!

Exercise

• Using a pipe: combine the faa files. How many sequences do we have in total?
• Using a pipe: combine the faa files. Extract only the header and then count how many duplicates do we have?
• Same as above, but how many sequences are not duplicated? Check the manual for uniq on what option allows you to do this.

# Question 1
cat downloads/*faa | grep -c ">"

# Question 2
cat downloads/*faa | grep ">" | sort | uniq -d | wc -l

# Question 3
cat downloads/*faa | grep ">" | sort | uniq -u | wc -l

find: Locate files based on conditions

find searches through the directories for files and directories with a given name, date, size, or any other attribute you care to specify.

# Find all files that end with txt, in downloads and all its subfolders
find downloads/ -name "*txt"

# Find files larger than 1 MB and show them as a long list
find downloads/ -size +1M -ls

sed: Manipulating the content of files

Sed is a stream editor that can be used to perform basic text transformations on an input stream (a file or input from a pipeline).

The most basic pattern to use sed for is sed ‘s/find/replace/’ file.

Basics: Search and replace

Let’s go back to our text files and assume that instead of writing Ex we want to be more clear and write Experiment, we can do this as follows:

sed 's/Ex/Experiment/' downloads/Experiment1.txt 

When doing this, we see that:

• Ex only gets replaced the first time it appears in a line, ie. the #Ex1_all_good in the first line gets not replaced
• sed will not overwrite the input file (unless you tell it to do so) but it will print the changed to the screen. If we wanted to save this in a file, we would need to re-direct the output to a new file.

To change all instances of Ex, we can do the following:

sed 's/Ex/Experiment/g' downloads/Experiment1.txt 

Sed patterns are regular expressions. For example, to replace control1 and control2 with control:

sed 's/control[0-9]/control/g' downloads/Experiment1.txt 

One important thing to remember is that certain symbols have specific meanings in UNIX. Examples are: comma, brackets, pipes. Do search for these in files, we need to escape them with a \. For example, we could replace the square with round brackets as follows:

# Replace the square brackets with round brackets
sed 's/N\[0.4uM\]/N(0.4uM)/g' downloads/Experiment1.txt 

Exercise

• In downloads/PF00900.faa , replace the GCA_ with Genome_
• In downloads/PF00900.faa , use a regular expression to replace the string of numbers after the GCA with a hello

# Question 1
sed 's/GCA_/Genome_/g'  downloads/PF00900.faa

# Question 2
sed 's/GCA_[0-9]*/GCA_hello/g'  downloads/PF00900.faa 

Removing lines and patterns

Sed can also be used to remove things, such as line(s) or patterns. Below is a list of examples that you might encounter or find useful.

# Remove the first line
sed '1d' downloads/Experiment1.txt

# Remove the last line
sed '$d' downloads/Experiment1.txt

# Remove lines 2-4
sed '2,4d' downloads/Experiment1.txt

# Remove all lines except lines 2-4
sed '2,4!d' downloads/Experiment1.txt

# Remove the first and the last line
sed '1d;$d' downloads/Experiment1.txt

# Remove lines that begin with an L
sed '/^L/d' downloads/Experiment1.txt

# Remove lines that end with a d
sed '/d$/d' downloads/Experiment1.txt

# Remove lines that end with a d or a D
sed '/[dD]$/d' downloads/Experiment1.txt

# Remove empty lines ('^$' means: the line ends right after it begins)
sed '/^$/d' downloads/Experiment1.txt

# Remove lines that begin with a capital letter
sed '/^[A-Z]/d' downloads/Experiment1.txt

# Remove lines that contain the pattern Ex
sed '/Ex/d' downloads/Experiment1.txt

# Remove lines that contain the pattern control or uM
# (\| for "or" works in GNU sed, as on Linux and Crunchomics, but not in the default sed on a Mac)
sed '/control\|uM/d' downloads/Experiment1.txt

# Remove the 2nd o in each line
sed 's/o//2' downloads/Experiment1.txt

# Remove all digits
sed 's/[0-9]//g' downloads/Experiment1.txt

# Remove all letters and digits
sed 's/[a-zA-Z0-9]//g' downloads/Experiment1.txt

# Remove the letter e, both small and capital
sed 's/[eE]//g' downloads/Experiment1.txt

For variables, loops, access rights, screen and cleaning up files from Windows, see Shell features and long jobs.