Bash scripting

Guides

The main place for my own scripts is /vm/video/scripts and /ssa/TV/scripts, or now linux/scripts on Sigillo and backups. For examples of bash scripts, see sigillo:/usr/share/doc/bash/examples.

Shell Script Frontend Tools

The package ssft has these functions:

  • ssft_display_message
  • ssft_display_error
  • ssft_file_selection
  • ssft_progress_bar
  • ssft_read_string
  • ssft_read_string (defaults)
  • ssft_read_password
  • ssft_select_multiple
  • ssft_select_multiple (defaults)
  • ssft_select_single
  • ssft_select_single (defaults)
  • ssft_show_file
  • ssft_yesno

For usage, see /usr/share/doc/ssft/examples/ssft-test.sh and run it for a demo -- very cool and simple.

Display steps and variables

bash -v ./test
bash -x ./test

See exit status

In a directory where there's a file called x, but no file called xx:

# test -f xx
# echo $?
1
# test -f x
# echo $?
0

#!/bin/bash [ -f /tmp/MYPROG ] && rm /tmp/MYPROG [ -f /tmp/MYPROG.ERR ] && rm /tmp/MYPROG.ERR echo $$ >/tmp/MYPROG #Do something here # # echo $? >/tmp/MYPROG.ERR exit

Rename files (source)

You want to rename all files in a directory by adding a new extension. Try to use the xargs command:

ls | xargs -t -i mv {} {}.old

xargs reads each item from the ls ouput and executes the mv command. The '-i' option tells xargs to replace '{}' with the name of each item. The '-t' option instructs xargs to print the command before executing it.

Find some part of a string using 'cut'

The tv capture scripts (channel, rsync-nfs, and others) need a sorted list of drives called 'tv'-something. They use the -d switch to indicate slash is the divider for finding the fourth column:

for i in $(df | cut -d'/' -f4 | grep tv | sort); do

These scripts also need to tell the user how much space is left on a drive, and use the -b switch

"($(df -h /$DRV | grep $DRV | cut -b 35-38)B free)"
We can also get a more precise measure of the space left, as now used in rsync-cc:
if [ $DISK = "tvspare" -o $(df /$DISK | grep $DISK | cut -b 32-40) -lt 10000 ]
This lets you skip drives have less than 10,000 bytes on them -- in practice newly formatted drives.

Find one of two strings

It's often useful to be able to check whether at least one of a list of strings is present in a file. This is accomplished with egrep, or grep -E, and an extended regular expression. For instance, this from rsync-nfs:

# Delete the list if it contains no avi or txt files (add extensions as needed)
if [ "$(egrep '(txt|avi)' /tmp/rsync-$BOX-files)" = "" ] ; then rm /tmp/rsync-$BOX-files ; fi
While I've not tested it, it's likely we can also use '(txt|avi|mp4)'.

To find a string that starts with a hyphen,
grep '\-lt' *

For loop

for file in [jx]* ; do
rm -f $file # Removes only files beginning with "j" or "x" in $PWD.
echo "Removed file \"$file\"".
done

Use a list of strings to operate on in turn

for box in {brazzi,lucca,pisa,prato} ; do ssh $box "sudo cp /etc/network/interfaces.ATS /etc/network/interfaces" done

Remove blank lines

sed:

$ sed '/^$/d' input.txt > output.txt

grep:

$ grep -v '^$' input.txt > output.txt

Remove line feeds

tr -d '\012' zap2it-nolinefeed
tr -d '\n' <50zap > 1zap
sed '/search/{N;s/\n//;}' file

Add a line feed before a string

sed 's/ test

Add a line feed before a paragraph

sed 's/

/\n

/g' $FIL > $FIL.spaced

Insert text at the first line of a file

sed '1i\
your text goes here' file_name > new_filename

Append text on the last line of the file

sed '$a\
your text goes here' file_name > new_filename

Double condition in a while loop

You can make a while loop depend on a string of conditions linked by && -- this one is from the blue-serve script. I don't know how to make the countdown seconds refresh while the rest remains, so I clear the page each time!

clear; n=1
while [ -z "$(ifconfig -a | grep bnep0)" ] && [ $n -lt 31 ] ; do
echo -e "\v\v\v\v\v\v\v\v\v\v\v\t\tThe bluetooth server is running"
echo -e "\t\tPlease tell the client to connect ... $(( 31-$n ))"
echo -n -e "\t\tOr press Ctrl-c to quit"
sleep 1; clear; let n=$(( $n+1 ))
done

This one says, "if there's no bnep0 device, check that we haven't waited more than thirty seconds" -- on the other hand, if the bnep0 device is found, forget about the wait time and exit the while loop right away.

Double conditions in an if statement

Test if either a OR b is true:

if [ "$r" = "Y" -o "$r" = "y" ]
Test if both a AND b are true:
if [ "$r" = "Y" -a "$r" = "y" ]
Combine:
if [ $SCORE -gt 53 -a $MP3SIZE -gt 45000000 -o $SCORE -gt 54 ]

File size

ls -s

wc -c

#!/bin/bash
FILENAME=/home/heiko/dummy/packages.txt
FILESIZE=$(stat -c%s "$FILENAME")
echo "Size of $FILENAME = $FILESIZE bytes."

ls -l filename  |awk -F" "'{ print $5 }'

du -h file
du -k file

X=~/The_File
size=$(du -b ${X} | sed 's/\([0-9]*\)\(.*\)/\1/')

Querying variables (see bash tests)

Test if the user typed in y or Y:

if [[ "$r" = [Yy] ]]

Test if a variable has not been assigned:

DRIVER="$1"
if [ -z "$DRIVER" ]; then
  echo "Driver name must be specified on the command line."
  exit 1
fi

Nice, no? Similarly, test if a string has some content:

if [ "$(grep natsemi /proc/modules)" != "" ]
if [ -n "$(grep ipw2200 /proc/modules)" ]

Or if it's empty:

if [ "$(grep natsemi /proc/modules)" = "" ]
if [ -z "$(grep natsemi /proc/modules)" ]

This one checks if a file exists (cf. tv-move and dv2xvid-mencoder):

if [ -e $TAR/.mounted ]
if [ -e $VCODEC-2 ]
This one checks if the file doesn't exist:
if [ ! -e "$file" ] # Check if file exists.
Check if a file exists and is not empty (see rsync-nfs):
if [ -s  /tmp/rsync-$BOX-files ]

Useful if the file is created by a script and may be empty.

Test if a variable is an integer (finally an elegant solution):
if [ "$(echo "$var" | egrep -s '^[-+]?[0-9]+$' )" ] ; then echo "this is a number" ; fi
if if [ "$(echo "$1" | egrep '^[0-9]+$' )" ] ; then echo "this is a positive integer" ; fi
Or define a function (not tested):
function is_integer {
  [[ $1 = ?([+-])+([0-9]) ]]
}
Test if a file contains a string:
if grep -q string file
 then echo "File contains at least one occurrence of string"
fi
Test if a string contains a letter-sequence:
word=Linux
letter_sequence=inu
if echo "$word" | grep -q "$letter_sequence" # The "-q" option to grep suppresses output
then echo "$letter_sequence found in $word"
else echo "$letter_sequence not found in $word"
fi

Directory tests

Tests if a directory exists (cf. Bash FAQ):

if [ -d $DIR ]
 then echo hi
fi

This one tests if a directory doesn't exist:

if test ! -d $DIR
 then echo hi
fi

Test if a directory has files of a certain type in it -- note the ! to negate:

if [ ! "$(ls *.txt 2> /dev/null)" ]
 then echo -e "\nNo text files found.\n"
 else echo -e "\nText file found.\n"
fi

List only the directories

ls -p1 | grep \/

Or this one might work in some cases (not tested):

find 2* -type d

Which directory am I in?

pwd | sed 's/.*\///'

Useful in a loop that wants only directories:

for DIR in $(ls -p1 | grep \/) ; do

Arithmetric operations

Add two numbers:
let "LAST = $NUM_FILES - 1"
n=$[n+1]
let n=$(( $n+33 ))

Another arithmetric expression example:

X=`expr 3 \* 2 + 4`

Round up a decimal to a round number ending in zero:

INT=$(printf "%1.0f" $1) NEAREST=${2:-10} while ((INT % NEAREST)); do let INT++ done echo $INT
Comparisons

Then you have the comparisons (see http://www.faqs.org/docs/abs/HTML/comparison-ops.html):

if [ $TARGET -gt $(date +%s) ]
if [ $vt -gt 3 ]
if [ $s -eq 12 ]
if [ $LOOP -lt 0 ]; then LOOP=-1; fi

And a different type of comparison, from toMP3:

if [ "$2" = -d ]; then

Compare decimals

Or this weird one (see dv2divx):

if [ $VCODEC* != *-2 ]

And equally, from dv2vob,

if [ $# -lt 2 ]
An odd one from ABSG:
if echo "Next *if* is part of the comparison for the first *if*."
 if [[ $comparison = "integer" ]]
  then (( a < b ))
  else [[ $a < $b ]]
 fi
  then echo '$a is less than $b'
fi
Get a file's basename, extension, and directory name (from bash_faq; see also http://splike.com/wiki/Bash_Scripting_FAQ):
# set the 'file' variable first

# get extension; everything after last '.'
ext=${file##*.}
# For instance, list only the file extension in a directory:
for file in *.*
  do
   echo ${file##*.}
done
exit
# basename
basename=`basename "$file"`
# everything after last '/'
basename=${file##*/}

# stem

# dirname
dirname=`dirname "$file"`
# everything before last '/'
basename=${file%/*}
Reading from files
Pick each word in sequence from a file:
for word in `cat somefile`
do
    echo Do something with $word here.
done
Read a line at a time from a file:
#!/bin/bash
echo enter file name
read fname
  
exec<$fname
value=0
while read line
do
  value=`expr $value + 1`;
  echo $value && echo $line
done
while read myline do echo $myline done < inputfile

See dcl3's avi2flash-list for a case where you have to use exec 9 (cf. http://anton.lr2.com/archives/2005/03/23/read-a-file-with-bash/ 12-14).

Read a line by number (see /usr/local/bin/insert-timecodes-from-file):
for i in $(seq 1 `wc -l $captions | cut -d" " -f1`) ; do
  n=$[n+1]
  if [ "$n" -lt "5" ]
   then sed -n "$i"p $captions >> "$captions"-fixed
   else sed -n "$i"p $captions >> "$captions"-fixed
    m=$[m+1]
    n=0
    sed -n "$m"p $timecodes >> "$captions"-fixed
  fi
done

User input
for x
  do
  echo "Type 'yes' to continue..."
  read response

  if [[ "$response" = [Yy][Ee][Ss] ]]; then
    kill `ps aux | grep $x | grep -v grep | grep -v "skill $x" | awk
    '{print $2}'`
  fi

  if [[ "$response" != [Yy][Ee][Ss] ]]; then
    echo "Exiting..."
  fi

done

Wait five seconds for the user to press any key once, unechoed (see read under man bash):

read -t 5 -n 1 -s
Switchbox
Set the port in a switchbox
case $2 in
  5003) vlc $1 -I dummy --sout '#std{access=mmsh,dst=:5003}' --ttl 12 --loop;;
  5900) vlc $1 -I dummy --sout '#std{access=mmsh,dst=:5900}' --ttl 12 --loop;;
  8080) vlc $1 -I dummy --sout '#std{access=mmsh,dst=:8080}' --ttl 12 --loop;;
  *)  echo "Sorry, that port is not available -- use 5003, 5900, or 8080" && exit;;
esac

Brace expansion (see)

echo sp{el,il,al}l

Generate all numbers between 1 and 99:

echo {0,1,2,3,4,5,6,7,8,9}{1,2,3,4,5,6,7,8,9}

Arrays

# Load the filenames into an array
declare -a FILES
FILES=( `find $DIR | grep txt | tr '\n' ' '` )

# Number of files
NUM_FILES=${#FILES[*]}

let "LAST = $NUM_FILES - 1"

for ((i = 0 ; i < $LAST ; i++ ))
do
# ...............
done
Clear an array element

unset STARTDROP[${i}]

Clear an array

unset STARTDROP

See also the soundmerge script.

Functions

List names without extensions:
#!/bin/bash

function stemname()
{
  local name=${1##*/}
  local name0="${name%.*}"
  echo "${name0:-$name}"
}

for i in *
  do
   stemname $i
done

List extensions only:

function ext()
{
  local name=${1##*/}
  local name0="${name%.*}"
  local ext=${name0:+${name#$name0}}
  echo "${ext:-.}"
}

for i in *.*
  do
   ext $i
done

Nice use of functions, though I don't understand how they work.

Get the PID of some program

ps ax | grep hotkeys | grep -v grep | awk '{ print $1 }'

-- where "hotkeys" is the name of the program you want the pid for. Or put it straight into a variable, using this alias in your bash shell:

pid=$(ps -ax | grep $1 | grep -v grep | awk '{ print $1 }')

Then you can do stuff like "kill -HUP $pid" to reread config files.

dates

Read about the secret of relative dates in "info coreutils date" -- for instance,

YESTERDAY=echo $(date --date="-1 day" +%Y-%m-%d)
This can be pushed quite far, as in the tape-timestamp script, which adds running seconds to any past time:
START=$(date +%s)
LAPSED=$[$(date +%s)-$START]
echo $(date -d "+$LAPSED seconds"\ $YEAR-$MONTH-$DAY\ $HOUR:$MIN +%F\ %H:%M:%S)
Or add a number of seconds to the current time:
echo $(date -d "+$LAPSED seconds"\ $(date +%F)\ $(date +%H:%M:%S) +%F_%H:%M:%S)

Perhaps more usefully, convert a duration in seconds to a duration in hours and minutes:

echo $(date -d "+$LAPSED seconds"\ $(date +%F) +%H\ hours,\ %M\ minutes,\ and\ %S\ seconds)

Convert hours and minutes to seconds for arithmetric operations (this uses UTC):

CUT1="$(echo $(date -u -d 1970-01-01\ 00:28:46 +%s))"
Back to hours and minutes:
echo $(date -u -d "+$CUT1 seconds"\ $(date +%F) +%M:%S)
Reconstruct a base directory address from a filename (no initial or final slash):
DIR="$(echo $FIL | sed -r 's/([0-9]{4})-([0-9]{2})-([0-9]{2}).*/\1\/\1-\2\/\1-\2-\3/')"
Reconstruct a base directory address from a date (includes initial and final slash):
DIR="$(echo $DATE | sed -r 's/([0-9]{4})-([0-9]{2})-([0-9]{2})/\/\1\/\1-\2\/\1-\2-\3\//')"
Note that the "-r" (regex) switch doesn't work with the sed on OSX; use gsed.

Remove the middle of a string -- any content, fixed number of characters -- use (.{n}) -- which makes sed into a fancy cut:

echo " 422198868 2007-05-01 14:06 2007-05-01_1300_MSNBC_Tucker_Carlson.avi" | sed -r 's/([0-9\ ]{11})(.{17})(.*)/\1\3/'

Switch two strings around (keep the initial tab for the first, but not the second), and leave out the middle strings (used with check-cc-backlog to generate a tab-delimited file of all word counts):

echo " 4324 Total number of words on 2007-07-10" | sed -r s/'([0-9\t]{5}).*([0-9-]{10})/\2\1/'
2007-07-10 4324

Suppress output selectively

Redirect the error message (idenfied by '2'), but not the value (identified by '1'):

SCCORE2="$( ssh $1 "cat /$2/$DDIR/$i 2> /dev/null" )"
./example.pl > /dev/null 2>&1

The second example redirects both.

cmp -- Compare two files

if cmp a b &> /dev/null  # Suppress output.
  then echo "Files a and b are identical."
  else echo "Files a and b differ."
fi


sed

Replacements using variables -- don't quote:

TAV="Tavis_Smiley"
TAV2="TavisSmiley"
sed s/KCET_$TAV/$TAV2/ filename

Replace only matches on lines starting with 200

sed '/200/s/XXX/KCET/' filename
Replace only matches on lines that contains an expanded variable $ID
sed /$ID/s/Tavis/Lavis/ filename
Replace the first twenty characters with $STEM in lines that start with 200:
sed s/^200................./$STEM/

Prepend the string Howdy to lines that start with a 2, or $STEM_ to lines that contain $PROG

sed -r 's/^(2)/Howdy\1/g'
sed 's/2/Howdy2/g'

sed /$PROG/s/^/$STEM\_/

Remove _$PROG from the end of lines that contain $PROG:

sed /$PROG/s/_$PROG'$'//

Pick the date or time out from a timestamp (from Andrey):
echo 2007-01-01_1930_KNBC_Access_Hollywood_2007-01-01_19:48:13 | sed -r 's/([0-9]{4}-[0-9]{2}-[0-9]{2}).*/\1/'
2007-01-01

echo 2007-01-01_1930_KNBC_Access_Hollywood_2007-01-01_19:48:13 | sed -r s/'.*([0-9]{2}):([0-9]{2}):([0-9]{2})/\1:\2:\3/'
19:48:13

Add a string inside a filename:

echo "2007-01-17_0000_MSNBC_Chris_Matthews000357.png" | sed -r s/$NAM'([0]{3})'/$NAM'.img000'/
2007-01-17_0000_MSNBC_Chris_Matthews.img000357.png
Am I in a directory called z?

if [ "$(echo `pwd` | sed s/.*\\///)" != "z" ]

ed -- editing within a bash script

Within a bash script, use ed to make editing changes to a file:

ed filename
,s/old/new/g
w
q
-- the w writes and the q quits. You can practice ed from the command line.

ed gives you the number of characters in the file as feedback to a write command.

If you need to use a slash or space, escape it first:
,s/2005-04\//rm\ \/tv1\/2005\/2005-04\//g
replaces "2005-04/" with "rm /tv1/2005/2005-04/". A comma or a percent sign (%) are equivalent, meaning "operate on every line".

To match only the beginning of each line, use ^ to start the regular expression:
,s/^2005/rm\ \/tv1\/2005\/2005/g
In command-line mode, ^ means "see the current line and move to the previous" -- ed starts at the bottom of a file.

To incorporate ed into a bash script, you use the << label redirector as follows:
ed /tmp/$(date +%F)-deleted << EOF
%s/^$(date +%Y-%m)/rm\ \/ssa\/TV\/$YR\/$(date +%Y-%m)
w $LOG/$(date +%F)-deleted
q
EOF
The logic here is that the << EOF command tells ed to read commands from the bash script it is in until it encounters a line containing only EOF in that script. So you can give a series of editing commands that will be passed to ed rather than to bash.

Notes
  • at least under Debian's recent bash, you cannot simply write (w) to the file you read; you need to give it a new name or give some variant of the w command (which I don't know)
  • in the %s line, ed requires leaving the /g off, while ex wants it in
  • in the %s line, you cannot use variables that have slashes in them, such as $DIR in tv-move.

See man ed for details.

A more powerful editor is ex, the executable version of vim. Shinn wrote,
ex myfile < my.script
and my.script has
%s/old/new/g
wq!
This works, but the << EOF is useful if you don't want to place the script with the commands in a different file. Besides, ed has a man page that actually explains how to use it, while vim is so huge it's hard to even get into.

If you want to use ex instead of ed, you need to add a /g at the end of the %s/ line -- the rest should be the same, though in practice I couldn't make ex work.

screen

Issue this for some pretty cool stuff:

man screen

Also see if there's bitchx and nohup on your systems.

"The program screen is a standard GNU Unix utility that totally rocks. It is great for reestablishing a CLI session. I beam into a machine from home; use screen and disconnect; beam back into the machine from work and can continue on."

Process managment

  • The process ID (PID) of the current process is $$
  • The PID of the most recent backgrounded process is $!
  • For an example of use, see /vc/video/scripts/channel and ccap

Working remotely

cp -R <filename> <destination> >& zz &

works fine -- but use -v for verbose, as zz is otherwise empty.
Note that cp also has a -u switch for update only!

Passing arguments from the input line

 # this script predicts the weather in Texas
clear
echo "This is your argumentative Texas forecast for: "
date
echo "$1 is expecting heavy $2 today, while $3 will be $4."

Source

Managing debian packages

Various fancy ways to pick out which packages you want -- here's one that claims to upgrade some specific subset of packages; I don't understand how it works:
dpkg --get-selections | grep -w install | awk '{print $1}' | xargs -m 10 -- apt-get -y --reinstall install

See also Commands for some examples.

  

 

top
Debate
Evolution
CogSci

Maintained by Francis F. Steen, Communication Studies, University of California Los Angeles


CogWeb