Sunday, 25 May 2014

Adding command line options to your shell scripts


You have a new data set, which has a number of samples in it which need to be processed. You choose one sample to experiment with, and step by step you figure out a suitable list of commands to go from raw data to final result. You realise you have to do this analysis for every sample now, so choosing some way to automate the process is now the sensible thing to do.

There are many different ways to automate a set of commands in Unix. They can be full-blown workflow engines (Taverna, Kepler), pipeline systems (BPIPE, Ruffus, Rubra), dependency methods (make, Ant, Maven), or a scripting language (Bash, Python, Perl).

The oldest and arguably simplest method is to use a shell script. All this really means is putting the series of commands you need to run into a text file, and telling the shell to run them all. Although some may baulk at the idea of a shell script, they have the advantage of being very portable and lightweight, and have many modern features that people don't realise. I will assume the use of the BASH shell, as it is available on most Unix environments, and is default on Linux and MacOSX.

A shell script

Type the following into a text editor and save it with the name "biox":

echo -n "Hello $USER, today is "
date +%A

You've created a BASH script called "biox" which is meant to print a greeting to the screen.

Running it

Let's try and run our new scripot:

% biox
biox: command not found

That didn't work. Ah, that's right, the current directory isn't in my PATH, so I need to refer to it directly:

% ./biox
bash: ./biox: Permission denied 

That didn't work either. Ah yes, only files with "execute" permission can be run:

% chmod +x biox
% ./biox
Hello torsten, today is Sunday

Success! An alternative way to run it is to pass it directly to the BASH command as an input script:

% bash biox
Hello torsten, today is Sunday

The advantage of making it executable is that it looks and feels like a standard Unix command, and you don't have to remember what scripting language to run it with. The shell figures that out automatically for you from the magic first line #!/bin/bash of the script.

Script Parameters

The motivation for writing a shell script was to automate the same set of commands for many samples. This means we need some way to tell it which data files to run on each time, without having to create a separate script for each sample.

Imagine we have lots of genome sequence runs, each one for a different Staphylococcus isolate from a hospital MRSA outbreak. Each sample's data is a paired-end Illumina run conisting of two files: Strain_R1.fastq.gz and Strain_R2.fastq.gz. Below is a script called assemble with 4 parameters which will automate the assembly of a given isolate:

if [ $# -lt 4 ]; then
  echo "Usage: $0 outputFolder kmerValue Read1 Read2"
  exit 1
velveth "$DIR" "$KVALUE" -shortPaired -fmtAuto -separate "$LEFT" "$RIGHT"
velvetg "$DIR" -exp_cov auto -cov_cutoff auto -very_clean yes
echo "Results are in: $DIR" 

The first "if" block checks to see that we provided 4 parameters, and if not, it prints a usage message and exists. The next 4 lines copy the 4 positional command line parameters into permanent, more understandable variables. The next 2 lines run velveth and velvetg accordingly, using as flexible options as possible. The final line just prints a message stating where the results are.

% ./assemble
Usage: ./assemble outputFolder kmerValue Read1 Read2

% ./assemble MRSA_J19d 51 J19d_R1.fq.gz J19d_R1.fq.gz 
(lots of velveth output here)
(lots of velvetg output here)
Results are in: MRSA_J19d

It is easy to envisage modifying this script to include pre-steps like read clipping, filtering and stitching; and post-steps like gap-filling and scaffolding. The opportunities for automation are endless.

Script Options

Clearly the ability to pass positional command line parameters to your script turns it from a bunch of fixed commands into a more generic re-usable tool. To make it even more flexible however you need command line options. At this point, many people would give up on BASH and move to a scripting language such as Python or Perl. However you may be surprised to know that BASH has inbuilt support for the getopt style of command line options!

Below is a newer version of our assembly script. It only has 3 mandatory parameters now (folder, R1, R2). The kmerValue is now default to 51, but can be changed using the -k option. We also add a new option -c to set the coverage cutoff, which defaults to auto.

# set our defaults for the options
# parse the options
while getopts 'c:k:' opt ; do
  case $opt in
    c) CUTOFF=$OPTARG ;;
    k) KVALUE=$OPTARG ;;
# skip over the processed options
shift $((OPTIND-1)) 
# check for mandatory positional parameters
if [ $# -lt 3 ]; then
  echo "Usage: $0 [options] outputFolder Read1 Read2"
  echo "Options: -k kmervalue (def: $KVALUE) | -c cov_cutoff (def: $CUTOFF)"
  exit 1
# do the assembly
echo "Assembling with K=$KVALUE and cutoff=$CUTOFF"
velveth "$DIR" "$KVALUE" -shortPaired -fmtAuto -separate "$LEFT" "$RIGHT"
velvetg "$DIR" -exp_cov auto -cov_cutoff "$CUTOFF" -very_clean yes
echo "Results are in: $DIR" 

As you can see, adding command line options requires little effort, and makes the script more flexible and professional.

% ./assemble
Usage: ./assemble [options] outputFolder Read1 Read2"
Options: -k kmervalue (def: 51) | -c cov_cutoff (def: auto)"

% ./assemble -k 83 -c 3 MRSA_J19d J19d_R1.fq.gz J19d_R1.fq.gz 
Assembling with K=83 and cutoff=3
(lots of velveth output here)
(lots of velvetg output here)
Results are in: MRSA_J19d

Error handling

The example scripts in this post only do a small amount error checking - they ensure the correct number of command line parameters were provided, but that is it. It doesn't check that all the options and parameters were valid eg. that the kmer value was an integer and not too small or too big, or that the read files actually existed and were readable, or that the output directory existed already or not. Neither did it check that the velveth and velvetg commands successfully ran or not. Nor did we catch the situation where the user pressed CTRL-C in the middle of our script!

The simplest thing you can do is add the following line to the top of your script:

set -e

This will cause the script to exit/fail if any of the commands that are run within the script fail (ie. they return a non-zero exit code). More detailed error handling is beyond the scope of this article but it something you should consider when automating the processing of huge data sets.


BASH scripts are still useful in today's bioinformatics world. They are a simple, portable way to automate your analyses. Smart use of command line parameters and command line options will make them even more flexible and potentially shareable with other bioinformaticians. Add in some error handling and defensive programming and you will produce quality tools without the need for higher level or more advanced scripting systems.

Further reading


  1. That's a lot of useful information in one spot. Thanks Torsten!

    1. Thanks Greg. It's amazing how long you can use a tool and still find new features 20 years on!

  2. Amazing, this little article here submmarises a lot of code that could take me to figured out like 2 or 3 days reading the BASH documentation but here I read it in 10 minutes Thanks so much I really like your blog because the style is very simple.

    1. Thank you. I try to make the articles clear and useful for people.

  3. This comment has been removed by a blog administrator.

  4. Good day friends. The truth is that GREAT MOTHER IS REAL AND UNIQUELY POWERFUL. Contact the Great Mother today and she will help you. when my husband left me for another woman, i thought that was the end but i contacted many spell casters but there was no positive responses and everything seemed not to be working and i was losing my mind until my cousin told me about the GREAT MOTHER who also helped her to bring back her man and also helped her to cure her herpes disease. I gave it a try and i contacted the Great Mother on her Website and i saw various testimonies about her on her good works how she has helped a lot of people. I explained my problems to her. she laughed and told me that, everything will be okay again and that she will help me bring back my man in 3 days time. I actually thought that it was a joke but since i had no other options, i gave it a try and i did what she instructed me do and to my greatest surprise, my man came back to me begging in 3 days time just as she told me. Contact the Great Mother now on her website:   and you can also reach her on her email:  or you can equally contact her on her whatsapp number: +17025514367

  5. All thanks to Dr OLIHA for curing my herpes virus/hpv with his herbal medicine, i do not have much to say but with all my life i will forever be grateful to him and God Almighty for using Dr OLIHA to reach me when i thought it was all over, today i am happy with my life again after the medical doctor have confirmed my HERPES SIMPLEX VIRUS / HPV of 5 is gone,i have never in my life believed that HERPES SIMPLEX VIRUS could be cured by herbal medicine. so i want to use this means to reach other persons who have this disease by testifying the power of Dr OLIHA that all hope is not lost yet, try and contact him by any means for any kind of disease with his email: add him on whatsapp line or call +2349038382931.

  6. Hello Everyone out there,I am here to give my testimony about a Herbalist doctor who helped me . I was infected with HERPES SIMPLEX VIRUS in 2011, i went to many hospitals for cure but there was no solution, so I was thinking how can I get a solution out so that my body can be okay. One day I was in the river side thinking where I can go to get solution. so a lady walked to me telling me why am I so sad and i open up all to her telling her my problem, she told me that she can help me out, she introduce me to a doctor who uses herbal medication to cure HERPES SIMPLEX VIRUS and gave me his email, so i mail him. He told me all the things I need to do and also give me instructions to take, which I followed properly. Before I knew what is happening after two weeks the HERPES SIMPLEX VIRUS that was in my body got vanished . so if you are also heart broken and also need a help, you can also email him at {}
    or whatsapp him number: +2349038382931. or website is
    Contact him today and you will have a testimony...Good luck!

    Dr. OLIHA Also Cures:
    1. HIV/AIDS
    2. HERPES 1/2
    3. CANCER
    4. ALS (Lou Gehrig’s disease)
    5. Hepatitis B
    6. chronic pancreatic
    7. Emphysema
    8. COPD (Chronic Obstructive Pulmonary Disease