Removing non-alphanumeric characters with sed

39
votes

I am trying to validate some inputs to remove a set of characters. Only alphanumeric characters plus, period, underscore, hyphen are allowed. I've tested the regex expression [^\w.-] here http://gskinner.com/RegExr/ and it matches what I want removed so I not sure why sed is returning the opposite. What am I missing?

My end goal is to input "Â10.41.89.50 " and get "10.41.89.50".

I've tried:

echo "Â10.41.89.50 " | sed s/[^\w.-]//g returns Â...

echo "Â10.41.89.50 " | sed s/[\w.-]//g and echo "Â10.41.89.50 " | sed s/[\w^.-]//g returns Â10418950

I attempted the answer found here Skip/remove non-ascii character with sed but nothing was removed.

6
regexbashsedreplace
Try adding the -r option to sed so it will recognize extended regular expressions. – Barmar
sed doesn't understand the special character classes like \w. Just use [a-zA-Z0-9_-]. – Mark Reed
neither -r nor using [a-zA-Z0-9_-] works. Well echo "Â10.41.89.50 " | sed s/[a-zA-Z0-9.-]//g returned  but echo "Â10.41.89.50 " | sed s/[^a-zA-Z0-9.-]//g still returned Â10.41.89.50. – wanderingandy

6 Answers

60
votes

's -c (complement) flag may be an option 

echo "Â10.41.89.50-._ " | tr -cd '[:alnum:]._-'
19
votes

You might want to use the [:alpha:] class instead:

echo "Â10.41.89.50 " | sed "s/[[:alpha:].-]//g"

should work. If not, you might need to change your local settings.

On the other hand, if you only want to keep the digits, the hyphens and the period::

echo "Â10.41.89.50 " | sed "s/[^[:digit:].-]//g"

If your string is in a variable, you can use pure bash and parameter expansions for that:

$ dirty="Â10.41.89.50 "
$ clean=${dirty//[^[:digit:].-]/}
$ echo "$clean"
10.41.89.50

or

$ dirty="Â10.41.89.50 "
$ clean=${dirty//[[:alpha:]]/}
$ echo "$clean"
10.41.89.50

You can also have a look at 1_CR's answer.

6
votes

Well sed won't support unicode characters. Use perl instead:

> s="Â10.41.89.50 "
> perl -pe 's/[^\w.-]+//g' <<< "$s"
10.41.89.50
4
votes

To remove all characters except of alphanumeric and "-" use this code:

echo "a b-1_2" | sed "s/[^[:alnum:]-]//g"
0
votes
<`[[:alnum:]_.@]`

This worked just fine for me. It preserved all of the characters I specified for my purposes.

0
votes
s/[^[:alnum:]+._-]//g

removes anything other than alphanumeric and ".+_-" characters.

echo "Â10.41.89.50 +-_" | sed s/[^[:alnum:]+._-]//g
Â10.41.89.50+-_