0
votes

I have the following HTML:

<div class="dialog">
<div class="title title-with-sort-row">
    <h2>Description</h2>
    <div class="dialog-search-sort-bar">
    </div>
</div>
<div class="content"><div style="margin-right: 20px; margin-left: 30px;">
    <span class="description2">
        With “Antonia Polygon – Standard”, you have a figure that is unique in the Poser community. 
        She is made available under a Creative Commons License that gives endless opportunities for further development. 
        This figure was developed by a group of talented members of the Poser community in a thirty-month effort. 
        The result is a figure that has very good bending and morphing behavior.
        <br />
    </span>
</div>
</div>

I need to find this div out of several divs of class="dialog", then pull out the text in span class="description2".

When I use the code:

description = soup.find(text = re.compile('Description'))
if description != None:
    someEl = description.parent
    parent1 = someEl.parent
    parent2 = parent1.parent
    description = parent2.find('span', {'class' : 'description2'})
    print 'Description: ' + str(description)

I get:

<span class="description2">
    With “Antonia Polygon – Standard”, you have a figure that is unique in the Poser community. 
    She is made available under a Creative Commons License that gives endless opportunities for further development. 
    This figure was developed by a group of talented members of the Poser community in a thirty-month effort. 
    The result is a figure that has very good bending and morphing behavior.
    <br/>
</span>

If I try to get just the text, without the HTML & non-ASCII chars, using

description = description.get_text()

I get a (UnicodeEncodeError): 'ascii' codex can't encode character u'\x93'

How do I convert this block of HTML to straight ascii?

1
The character “ is not an ASCII character. Is your goal to identify the most similar character that is ASCII (that's hard), or is your goal to simply remove all non-ASCII characters? Or is what you really want to output proper Unicode, e.g. UTF-8, rather than ASCII? - jogojapan
Just to remove all non-ascii chars - Stephen
Obligatory: bit.ly/unipain - Daenyth

1 Answers

2
votes
#!/usr/bin/env python
# -*- coding: utf-8 -*-

foo = u'With “Antonia Polygon – Standard”, you have a figure that is unique in the Poser community.She is made available under a Creative Commons License that gives endless opportunities for further development. This figure was developed by a group of talented members of the Poser community in a thirty-month effort. The result is a figure that has very good bending and morphing behavior.'

print foo.encode('ascii', 'ignore')

Three things to notice.

First is the 'ignore' parameter to the encode method. It instructs the method to drop characters that aren't in range of the selected encoding (in this case, ascii to be safe).

Second is that we're explicitly setting foo to be unicode by prepending the string with a u.

Third is the explicit file encoding directive: # -*- coding: utf8 -*-.

Also, if you don't read Daenyth's very good points in comments attached to this answer then you are a silly clod. xmlcharrefreplace can be used for great justice in the stead of ignore above if the output is to be used in HTML/XML.