How to rsync file pattern in subfolders

Question

I have the following remote folder/file structure:

/home/user/backup/siteA/regionA/data.sql
/home/user/backup/siteA/regionA/data.clean.sql
/home/user/backup/siteA/regionA/data.lite.sql
/home/user/backup/siteA/regionA/files/
/home/user/backup/siteA/regionB/data.sql
/home/user/backup/siteA/regionB/data.clean.sql
/home/user/backup/siteA/regionB/data.lite.sql
/home/user/backup/siteA/regionB/files/
/home/user/backup/siteA/regionC/data.sql
/home/user/backup/siteA/regionC/data.clean.sql
/home/user/backup/siteA/regionC/data.lite.sql
/home/user/backup/siteA/regionC/files/
/home/user/backup/siteA/regionD/data.sql
/home/user/backup/siteA/regionD/data.clean.sql
/home/user/backup/siteA/regionD/data.lite.sql
/home/user/backup/siteA/regionD/files/
/home/user/backup/siteB/regionA/data.sql
/home/user/backup/siteB/regionA/data.clean.sql
/home/user/backup/siteB/regionA/data.lite.sql
/home/user/backup/siteB/regionA/files/
/home/user/backup/siteB/regionB/data.sql
/home/user/backup/siteB/regionB/data.clean.sql
/home/user/backup/siteB/regionB/data.lite.sql
/home/user/backup/siteB/regionB/files/
/home/user/backup/a-script.sh
/home/user/backup/backup.log
/home/user/backup/siteC
...
...

I need to rsync *.clean.sql and *.lite.sql from siteA to a similar structure locally, eg:

/var/data/dbs/siteA/regionA/data.clean.sql
/var/data/dbs/siteA/regionA/data.lite.sql

I cant seem to get rsync to include/exclude the right things. I am trying:

/usr/bin/rsync --archive --del --verbose --verbose --compress --progress --exclude-from=excludes.txt --rsh='ssh -q' --dry-run user@remote:/home/user/backup/siteA /var/data/dbs/siteA

And my excludes.txt contains:

+ *.clean.sql
+ *.lite.sql
- *

However I just get:

receiving incremental file list
[sender] hiding directory siteA because of pattern *

I have a script which can do this on a per-site-region basis, but not as a "whole". I've tried adding stars and slashes to the excludes list, but cant seem to find the right combo. Any suggestions?

Update

I've got better results with

/usr/bin/rsync --archive --del --compress --progress --exclude-from=excludes.txt --rsh='ssh -q' user@remote:/home/user/backup/siteA /var/data/dbs/siteA

with the excludes.txt

+ */
+ *.clean.sql
+ *.lite.sql
- *

However, that also parses the contents of all the files folders (albeit to skip/exclude them) which slows it down a lot (and pollutes the rsync output with a huge directory listing).

Is there a "maxdepth" parameter?!

Stephane Rouberol Stephane Rouberol · Accepted Answer · 2012-08-08T12:19:19

How about:

cd /home/user/backup
find site* -name data.clean.sql -o -name data.lite.sql > filestobackup
rsync --archive --verbose --del --compress --progress \
--files-from=filestobackup -d \
--rsh='ssh -q' \
user@remote:/home/user/backup/ /var/data/dbs/

How to rsync file pattern in subfolders

Update

1 Answers