Problem With Regular Expression to Remove HTML Tags

Question

In my Ruby app, I've used the following method and regular expression to remove all HTML tags from a string:

str.gsub(/<\/?[^>]*>/,"")

This regular expression did just about all I was expecting it to, except it caused all quotation marks to be transformed into “ and all single quotes to be changed to ” .

What's the obvious thing I'm missing to convert the messy codes back into their proper characters?

Edit: The problem occurs with or without the Regular Expression, so it's clear my problem has nothing to do with it. My question now is how to deal with this formatting error and correct it. Thanks!

Answer 1

Use CGI::unescapeHTML after you perform your regular expression substitution:

CGI::unescapeHTML(str.gsub(/<\/?[^>]*>/,""))

See http://www.ruby-doc.org/core/classes/CGI.html#M000547

In the above code snippet, gsub removes all HTML tags. Then, unescapeHTML() reverts all HTML entities (such as <, &#8220) to their actual characters (<, quotes, etc.)

With respect to another post on this page, note that you will never ever be passed HTML such as

<tag attribute="<value>">2 + 3 < 6</tag>

(which is invalid HTML); what you may receive is, instead:

<tag attribute="&lt;value&gt;">2 + 3 &lt; 6</tag>

The call to gsub will transform the above to:

2 + 3 < 6

And unescapeHTML will finish the job:

2 + 3 < 6

Answer 2

You're going to run into more trouble when you see something like:

<doohickey name="<foobar>">

You'll want to apply something like:

gsub(/<[^<>]*>/, "")

...for as long as the pattern matches.

Answer 3

This regular expression did just about all I was expecting it to, except it caused all quotation marks to be transformed into “ and all single quotes to be changed to ” .

This doesn't sound as if the RegExp would be doing this. Are you sure it's different before?

See this question here for information about the problem, it has got an excellent answer:
Get non UTF-8 form fields as UTF-8 in php.

Answer 4

I've run into a similar problem with character changes, this happened when my code ran through another module that enforced UTF-8 encoding and then when it came back, I had a different file (slurped array of lines) on my hands.

Answer 5

You could use a multi-pass system to get the results you are looking for.

After running your regular expression, run an expression to convert &8220; to quotes and another to convert &8221; to single quotes.