Extract data from one file and save to another

Question

I'm running ansible playbook task using shell command to extract data from file based on regex match and save it to another file.

I tried using awk and sed but not able to get the regex working.

awk '$NF == "-m.comment.*\""' iptable.txt" > file1.txt

sed 's/\/.*' iptable.txt > file2.txt

I need to save any content from -m comment till the double quotes. to file1.txt and remaining content to file2.txt. If the line doesnot have comment field, then it should be saved only to file2.txt.

-P INPUT ACCEPT
-A INPUT -p tcp -m tcp --dport 80 -j ACCEPT
-A INPUT -p icmp -m state --state NEW -m comment --comment "Echo Request" -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m comment --comment "tcp" -j ACCEPT

Expected output: cat file1.txt

-m comment --comment "Echo Request"
-m comment --comment "tcp"

cat file2.txt

-P INPUT ACCEPT
-A INPUT -p tcp -m tcp --dport 80 -j ACCEPT
-A INPUT -p icmp -m state --state NEW -j ACCEPT
-A INPUT -p tcp -m state --state NEW -j ACCEPT

@JUSHJUSH I'm trying to extract till double quotes so I had double quotes as the matching character. Can you pls point me to right syntax for sed? — Knightfox
wrt I know regex is right as I tested with regex tester - proving that a regex works in some online tool ONLY proves that it works in that online tool. That does not mean it'll work in sed, awk or any other command line tool. There's just WAY too many regexp variants, tool variants, extensions, delimiters, options, etc. to think that you can test a regexp in foo and that means it'll work in bar. — Ed Morton

Ed Morton Ed Morton · Accepted Answer · 2019-08-30T14:25:27

With GNU awk for the 3rd arg to match():

awk 'match($0,/(.*)(-m.comment.*")(.*)/,a) {
    print a[2] " > foo"
    print a[1] a[3] " > bar"
}' file
-m comment --comment "Echo Request" > foo
-A INPUT -p icmp -m state --state NEW  -j ACCEPT > bar

With any awk:

awk 'match($0,/-m.comment.*"/) {
    print substr($0,RSTART,RLENGTH) " > foo"
    print substr($0,1,RSTART-1) substr($0,RSTART+RLENGTH) " > bar"
}' file

Just change " > foo" to > "foo" and ditto for bar to really write to new files.

If that's not exactly what you need then edit your question to clarify your requirements and provide more truly representative sample input/output.

Oh and when you wrote:

$NF == "-m.comment.*\""

There's nothing wrong with -m.comment.*\" as a regexp per-se BUT == is telling awk to do a literal string comparison, not a regexp comparison.

Given your updated question, just tweak the above to:

awk 'match($0,/(.*)(-m.comment.*")(.*)/,a) {
    print a[2] " > foo"
    print a[1] a[3] " > bar"
    next
}
{ print $0 " > bar" }' file
-P INPUT ACCEPT > bar
-A INPUT -p tcp -m tcp --dport 80 -j ACCEPT > bar
-m comment --comment "Echo Request" > foo
-A INPUT -p icmp -m state --state NEW  -j ACCEPT > bar
-m comment --comment "tcp" > foo
-A INPUT -p tcp -m state --state NEW  -j ACCEPT > bar

Extract data from one file and save to another

1 Answers