15
votes

The Story:

When you parse HTML with BeautifulSoup, class attribute is considered a multi-valued attribute and is handled in a special manner:

Remember that a single tag can have multiple values for its “class” attribute. When you search for a tag that matches a certain CSS class, you’re matching against any of its CSS classes.

Also, a quote from a built-in HTMLTreeBuilder used by BeautifulSoup as a base for other tree builder classes, like, for instance, HTMLParserTreeBuilder:

# The HTML standard defines these attributes as containing a
# space-separated list of values, not a single value. That is,
# class="foo bar" means that the 'class' attribute has two values,
# 'foo' and 'bar', not the single value 'foo bar'.  When we
# encounter one of these attributes, we will parse its value into
# a list of values if possible. Upon output, the list will be
# converted back into a string.

The Question:

How can I configure BeautifulSoup to handle class as a usual single-valued attribute? In other words, I don't want it to handle class specially and consider it a regular attribute.

FYI, here is one of the use-cases when it can be helpful:

What I've tried:

I've actually made it work by making a custom tree builder class and removing class from the list of specially-handled attributes:

from bs4.builder._htmlparser import HTMLParserTreeBuilder

class MyBuilder(HTMLParserTreeBuilder):
    def __init__(self):
        super(MyBuilder, self).__init__()

        # BeautifulSoup, please don't treat "class" specially
        self.cdata_list_attributes["*"].remove("class")


soup = BeautifulSoup(data, "html.parser", builder=MyBuilder())

What I don't like in this approach is that it is quite "unnatural" and "magical" involving importing "private" internal _htmlparser. I hope there is a simpler way.

NOTE: I want to save all other HTML parsing related features, meaning I don't want to parse HTML with "xml"-only features (which could've been another workaround).

2
I thought it was a bug when I saw your avatar under a beautifulsoup question with no answer and then I realized you asked the question! I can't help you everything I tried didn't work or involved two iterations. - dstudeba
I do not know how to do this, but for the specific use case provided as example I provided a different answer (so I posted it there). It is simpler in my opinion but may not be sufficient for other use cases - rll
Using it as a css selector?. Maybe in that case the simplest option could be not to use a common class selector, but an attribute selector. Selector '.myclass' is just the same that '[class=~"myclass"]', but selector '[class="class"]' is an element whose "class" attribute value is exactly equal to "myclass" (not myclass in a space sepated list). - miguel-svq
@miguel-svq good point, CSS selectors might help here depending on the use case. If we take that use case from a linked question, there is a regular expression pattern applied to a class attribute - this is something we cannot really achieve with CSS selectors, but, we can use them to narrow down the search and then do some manual checks. Thank you! - alecxe

2 Answers

6
votes

What I don't like in this approach is that it is quite "unnatural" and "magical" involving importing "private" internal _htmlparser. I hope there is a simpler way.

Yes, you can import it from bs4.builder instead:

from bs4 import BeautifulSoup
from bs4.builder import HTMLParserTreeBuilder

class MyBuilder(HTMLParserTreeBuilder):
    def __init__(self):
        super(MyBuilder, self).__init__()
        # BeautifulSoup, please don't treat "class" as a list
        self.cdata_list_attributes["*"].remove("class")


soup = BeautifulSoup(data, "html.parser", builder=MyBuilder())

And if it's important enough that you don't want to repeat yourself, put the builder in its own module, and register it with register_treebuilders_from() so that it takes precedence.

2
votes

The class HTMLParserTreeBuilder is actually declared on the upper module _init__.py, so there is no need to import directly from the private submodule. That said I would do it the following way:

import re

from bs4 import BeautifulSoup
from bs4.builder import HTMLParserTreeBuilder

bb = HTMLParserTreeBuilder()
bb.cdata_list_attributes["*"].remove("class")

soup = BeautifulSoup(bs, "html.parser", builder=bb)
found_elements = soup.find_all(class_=re.compile(r"^name\-single name\d+$"))
print found_elements

It is basically the same as defining the class as in the OP (maybe a bit more explicit), but I don't think there is a better way to do it.