Why do bash parameter expansions cause an rsync command to operate differently?

2
votes

I am attempting to run an rsync command that will copy files to a new location. If I run the rsync command directly, without any parameter expansions on the command line, rsync does what I expect

$ rsync -amnv --include='lib/***' --include='arm-none-eabi/include/***' \
  --include='arm-none-eabi/lib/***' --include='*/' --exclude='*' \
  /tmp/from/ /tmp/to/

building file list ... done
created directory /tmp/to
./
arm-none-eabi/
arm-none-eabi/include/
arm-none-eabi/include/_ansi.h
...
arm-none-eabi/lib/
arm-none-eabi/lib/aprofile-validation.specs
arm-none-eabi/lib/aprofile-ve.specs
...
lib/
lib/gcc/
lib/gcc/arm-none-eabi/
lib/gcc/arm-none-eabi/4.9.2/
lib/gcc/arm-none-eabi/4.9.2/crtbegin.o
...

sent 49421 bytes  received 6363 bytes  10142.55 bytes/sec
total size is 423195472  speedup is 7586.32 (DRY RUN)

However, if I enclose the filter arguments in a variable, and invoke the command using that variable, different results are observed. rsync copies over a number of extra directories I do not expect:

$ FILTER="--include='lib/***' --include='arm-none-eabi/include/***' \
  --include='arm-none-eabi/lib/***' --include='*/' --exclude='*'"
$ rsync -amnv ${FILTER} /tmp/from/ /tmp/to/

building file list ... done
created directory /tmp/to
./
arm-none-eabi/
arm-none-eabi/bin/
arm-none-eabi/bin/ar
...
arm-none-eabi/include/
arm-none-eabi/include/_ansi.h
arm-none-eabi/include/_syslist.h
...
arm-none-eabi/lib/
arm-none-eabi/lib/aprofile-validation.specs
arm-none-eabi/lib/aprofile-ve.specs
...
bin/
bin/arm-none-eabi-addr2line
bin/arm-none-eabi-ar
...
lib/
lib/gcc/
lib/gcc/arm-none-eabi/
lib/gcc/arm-none-eabi/4.9.2/
lib/gcc/arm-none-eabi/4.9.2/crtbegin.o
...

sent 52471 bytes  received 6843 bytes  16946.86 bytes/sec
total size is 832859156  speedup is 14041.53 (DRY RUN)

If I echo the command that fails, it generates the exact command that succeeds. Copying the output, and running directly gives me the expected result.

There is obviously something I'm missing about how bash parameter expansion works. Can somebody please explain why the two different invocations produce different results?

2
bashrsync
The interior single quotes are not removed when you expand the FILTER variable. Add set -x above the rsync line to and compare the output from the manual run and the variable run to see what I mean. This is the sort of thing that mywiki.wooledge.org/BashFAQ/050 is all about. - Etan Reisner
Thanks, set -x provides a lot of insight into what is going on here. The wiki entry explains a lot. - Piklor

2 Answers

6
votes

The shell parses quotes before expanding variables, so putting quotes in a variable's value doesn't do what you expect -- by the time they're in place, it's too late for them to do anything useful. See BashFAQ #50: I'm trying to put a command in a variable, but the complex cases always fail! for more details.

In your case, it looks like the easiest way around this problem is to use an array rather than a plain text variable. This way, the quotes get parsed when the array is created, each "word" gets stored as a separate array element, and if you reference the variable properly (with double-quotes and [@]), the array elements get included in the command's argument list without any unwanted parsing:

filter=(--include='lib/***' --include='arm-none-eabi/include/***' \
  --include='arm-none-eabi/lib/***' --include='*/' --exclude='*')
rsync -amnv "${filter[@]}" /tmp/from/ /tmp/to/

Note that arrays are available in bash and zsh, but not all other POSIX-compatible shells. Also, I lowercased the filter variable name -- recommended practice to avoid colliding with the shell's special variables (which are all uppercase).

1
votes

I like to break the arguments onto separate lines, for convenience sake:

ROPTIONS=(
   -aNHXxEh
   --delete
   --fileflags
   --exclude-from=$EXCLUDELIST
   --delete-excluded
   --force-change
   --stats
   --protect-args
)

and then call it thusly:

rsync "${ROPTIONS[@]}" "$SOURCE" "$DESTINATION"