Whole array operations on bash arrays

Today I wanted to take all the elements of a bash array and append /** to each one. This was for a script that syncs selected photo albums to cloud storage using rclone. As you know, rclone syncs directories, so to limit the transfer to a subset of of the directories, I used an --include pattern. Say the albums are cny13 fiji13 misc13 under archived. Then the command required is:

rclone sync archived mydrive:Photos --include "{cny13/**,fiji13/**,misc13/**}"

So the question is how to get this from an array containing:

declare -a ALBUMS=(cny13 fiji13 misc13)

Of course, I could run a small loop where I append /** to each element and accumulate in another array. This does work and efficiency isn't really an issue. But I guess the old APL fan in me was awakened and I wondered if I could transform the whole array in one fell swoop.

I tried:

echo "${ALBUMS[@]}/**"

but this only got me:

cny13 fiji13 misc13/**

The clue was supplied in an online tutorial on bash arrays in which an example was shown of parameter substitution. This in fact works just as well on arrays element by element.

echo "${ALBUMS[@]/%/\/**}"

This substitutes the end of line with /** escaping the leading slash, so we get:

cny13/** fiji13/** misc13/**

I haven't explained how the , is inserted between items, but the entire script using another trick to create a join function shows it:

#!/bin/bash

function join { local IFS="$1"; shift; echo "$*"; }

declare -a ALBUMS=(cny13 fiji13 misc13)
declare -a includes=("${ALBUMS[@]/%/\/**}")

cd ~/Albums || exit 1
albums=$(join , ${includes[@]})

rclone sync archived mydrive:Photos --include "{$albums}"

bash ranges

This is not widely known, although it is adequately documented in the bash(1) man page thus:

A sequence expression takes the form {x..y[..incr]}, where x and y are either integers or single characters, and incr, an optional increment, is an integer. When integers are supplied, the expression expands to each number between x and y, inclusive. Supplied integers may be prefixed with 0 to force each term to have the same width. When either x or y begins with a zero, the shell attempts to force all generated terms to contain the same number of digits, zero-padding where necessary. When characters are supplied, the expression expands to each character lexicographically between x and y, inclusive, using the default C locale. Note that both x and y must be of the same type. When the increment is supplied, it is used as the difference between each term. The default increment is 1 or -1 as appropriate.

This means that you can expect these expressions to work:

$ echo {0..10}
0 1 2 3 4 5 6 7 8 9 10
$ echo {z..a}
z y x w v u t s r q p o n m l k j i h g f e d c b a
$ echo {01..20..2}
01 03 05 07 09 11 13 15 17 19
$ echo f{20..01..2}
f20 f18 f16 f14 f12 f10 f08 f06 f04 f02

In the last case, notice a prefix was prepended as you would expect. Suffixes work too, of course. -2 will also work. The direction is determined by the range.

Thus {01..20} is a more efficient alternative to using

$(seq -w 01 20)

Brace expansion happens early so no use storing the expression in a variable or having one of the start, end, or increment a variable.

In case you are still wondering what ranges are useful for, a typical use is in a for loop:

for month in {01..12}
do
...
done

How to get fall through behaviour in bash case statements

As many people know C has the famous (or infamous) fall through behaviour in switch statements. Is something similar available for bash?

Indeed, just look at the man page:

Using ;& in place of ;; causes execution to continue with the list associated with the next set of patterns. Using ;;& in place of ;; causes the shell to test the next pattern list in the statement, if any, and execute any associated list on a successful match.

In other words, where you would leave out the break; in C you would write ;& instead of ;; in bash. Hope you have a good use case for it (sorry for the pun).

Avoid calling basename or dirname by using parameter expansion

These short cuts have been documented under parameter expansion for a long time but are not well known because people are used to writing basename and dirname.

Instead of using:

file=$(basename "$path")

use

file=${path##*/}

Instead of using:

dir=$(dirname "$path")

use

dir=${path%/*}

The advantage is not calling basename as an extra process. Although basename might be implemented as a builtin in some shells.

See the linked page for further short cuts.

"$@" is a shell idiom

TL;DR: Use "$@" when you want to pass arguments unchanged to a function or program.

When you read the shell documentation you will see that there are two main ways to refer to all the arguments for passing to a function or program: $* and "$@". What is the difference? This test script will demonstrate it:

#!/bin/sh

testargs() {
       echo testargs: $# arguments

       showargs $*
       showargs "$@"
}

showargs() {
       echo showargs: $# arguments
       for i
       do
               echo $i
       done
}

testargs 1 2 3
testargs word 'quoted phrase' word

The result should be:

testargs: 3 arguments
showargs: 3 arguments
1
2
3
showargs: 3 arguments
1
2
3
testargs: 3 arguments
showargs: 4 arguments
word
quoted
phrase
word
showargs: 3 arguments
word
quoted phrase
word

As you can see, the difference is manifest when an argument has whitespace. "$@" preserves the arguments, not parsing it again to break up at whitespace in arguments. Think if it as an idiom meaning pass arguments verbatim.

Why would you ever use $* though? Here's a place where you shouldn't use "$@".

su -c "$*" user

If you were to use "$@" and it contained multiple arguments, only the first argument would be used by -c and the others would follow, causing a syntax error. This however means that if you want to pass arguments with whitespace to -c, you have to quote them and escape the quotes too.

Blacklist a command in bash

I have an alias called nf. Sometimes due to fat fingers I end up typing mf and this starts up Metafont, which is installed because I use TeX, then I would have to exit it. I got tired of this and added this alias to $HOME/.alias:

alias mf='echo "Use \\mf if you really want metafont"'

If I really want to run mf, which is rarely, I can type \mf at the command line as the \ stops alias expansion. Invocations from shell scripts and Makefiles are not affected as $HOME/.alias is only read in by interactive shells.

BTW, please do not use this technique to block shell users from executing certain commands by aliasing them to something else. It's trivial to bypass in just the way I've shown.

How to remove the last command line argument in a bash script

In a bash wrapper script I needed to pass a bunch of arguments to the program. No problem, I'll do that with "$@". Then I had a new requirement: if the last argument is of the form *.ppm, I want the stdout of the program to go to this file. But any previous arguments, i.e. options, should be passed to the program. So it boiled down to this:

if nth argument matches *.ppm
  program "1st arg" .. "n-1th arg" > "nth arg"
else
  program "arguments"

To get the last element of an array, you can do ${argv[-1]}. Oops, you cannot do this with ${@[-1]}. So we have to make a copy in a local variable first:

declare -a argv=("$@")
declare file=${argv[-1]}

But we still have to remove the last argument of $@. We can't set it to the empty string, it still exists and will be seen by the program as a empty string. No we can't alter $@, so we have to use unset on argv. But this doesn't work:

unset argv[-1]

Ok, so we have to get the index of the last argument. This is one less than the length of the array, which is ${#argv[@]} (the reason for [@] is the quirky bash syntax for referring to the whole array). So we have to use $(()) to do arithmetic.

declare argc=$((${#argv[@]}-1))

So, putting it all together, the code looks like this:

declare -a argv=("$@")
declare argc=$((${#argv[@]}-1))
declare file=${argv[$argc]}
unset argv[$argc]

Then in the if branch we can write:

program ... ${argv[@]} > "$file" 

Whew!


There is a similar idea in the pop operation here, where we replace the contents of argv with all but the last element, using the subarray notation.