Convert character entities to their unicode equivalents

Question

I have html encoded strings in a database, but many of the character entities are not just the standard & and <. Entities like “ and —. Unfortunately we need to feed this data into a flash based rss reader and flash doesn't read these entities, but they do read the unicode equivalent (ex “).

Using .Net 4.0, is there any utility method that will convert the html encoded string to use unicode encoded character entities?

Here is a better example of what I need. The db has html strings like: <p>John &amp; Sarah went to see $ldquo;Scream 4$rdquo;.</p> and what I need to output in the rss/xml document with in the <description> tag is: <p>John & Sarah went to see “Scream 4”.</p>

I'm using an XmlTextWriter to create the xml document from the database records similar to this example code http://www.dotnettutorials.com/tutorials/advanced/rss-feed-asp-net-csharp.aspx

So I need to replace all of the character entities within the html string from the db with their unicode equivilant because the flash based rss reader doesn't recognize any entities beyond the most common like &.

Answer 1

My first thought is, can your RSS reader accept the actual characters? If so, you can use HtmlDecode and feed it directly in.

If you do need to convert it to the numeric representations, you could parse out each entity, HtmlDecode it, and then cast it to an int to get the base-10 unicode value. Then re-insert it into the string.

EDIT: Here's some code to demonstrate what I mean (it is untested, but gets the idea across):

string input = "Something with — or other character entities.";
StringBuilder output = new StringBuilder(input.Length);

for (int i = 0; i < input.Length; i++)
{
    if (input[i] == '&')
    {
        int startOfEntity = i; // just for easier reading
        int endOfEntity = input.IndexOf(';', startOfEntity);
        string entity = input.Substring(startOfEntity, endOfEntity - startOfEntity);
        int unicodeNumber = (int)(HttpUtility.HtmlDecode(entity)[0]);
        output.Append("&#" + unicodeNumber + ";");
        i = endOfEntity; // continue parsing after the end of the entity
    }
    else
        output.Append(input[i]);
}

I may have an off-by-one error somewhere in there, but it should be close.

Answer 2

would HttpUtility.HtmlDecode work for you?

I realize it doesn't convert to unicode equivalent entities, but instead converts it to unicode. Is there a specific reason you want the unicode equivalent entities?

updated edit

---

        string test = "<p>John &amp; Sarah went to see &ldquo;Scream 4&rdquo;.</p>";
        string decode = HttpUtility.HtmlDecode(test);
        string encode = HttpUtility.HtmlEncode(decode);

        StringBuilder builder = new StringBuilder();
        foreach (char c in encode)
        {
            if ((int)c > 127)
            {
                builder.Append("&#");
                builder.Append((int)c);
                builder.Append(";");
            }
            else
            {
                builder.Append(c);
            }
        }
        string result = builder.ToString();

Answer 3

you can download a local copy of the appropriate HTML and/or XHTML DTDs from the W3C. Then set up an XmlResolver and use it to expand any entities found in the document.

You could use a regular expression to find/expand the entities, but that won't know anything about context (e.g., anything in a CDATA section shouldn't be expanded).

Answer 4

this might help you put input path in textbox

        try
        {
            FileInfo n = new FileInfo(textBox1.Text);
            string initContent = File.ReadAllText(textBox1.Text);
            int contentLength = initContent.Length;
            Match m;

            while ((m = Regex.Match(initContent, "[^a-zA-Z0-9<>/\\s(&#\\d+;)-]")).Value != String.Empty)
                initContent = initContent.Remove(m.Index, 1).Insert(m.Index, string.Format("&#{0};", (int)m.Value[0]));

            File.WriteAllText("outputpath", initContent);
        }

        catch (System.Exception excep)
        {

            MessageBox.Show(excep.Message);

        }



    }