scp: how to copy a file from remote server with a filter

Question

I am trying to use scp to copy large log files from a remote server. However I want only the lines in remote log file that has a string "Fail". This is how I am doing it currently

scp user@ip:remote_folder/logfile* /localfolder

This copies all the files starting with logfile in remote server to my local folder. The files are pretty large and I need to copy only the lines in those log file, containing the string "Fail" from remote server. Can any body tell me how to do this? Can I use cat or grep command?

A little tip: if you are transferring text files, then using the -C flag to ssh/scp to enable compression can make things much faster over a slow connection... — thkala

Finwood Finwood · Accepted Answer · 2016-01-14T12:49:05

Use grep on the remote machine and filter the output into file name and content:

#!/usr/bin/env bash

BASEDIR=~/temp/log

IFS=$'\n'
for match in `ssh user@ip grep -r Fail "remote_folder/logfile*"`
do
    IFS=: read file line <<< $match
    mkdir -p `dirname $BASEDIR/$file`
    echo $line >> $BASEDIR/$file
done

You might want to look at an explanation to IFS in combination with read.

scp: how to copy a file from remote server with a filter

2 Answers