Filtering function issues in BS4 find_all() statement in Python

Question

I'm scraping an HTML webpage. I'm using the BeautifulSoup lib (4.6.0) and Python (3.7) on a Mac (Sierra).

Among other things, I see a bunch of 'div' tags which have class attributes. Some of the 'div' tags come with multiple class attribute values. Now I want to filter based on tag name and class attribute values, e.g. I want to find the 'div' tags that have class='a' but not class='b' (and, yes, some of the div tags come with class='a b').

To get those tags I tried to use a filtering function as mention in the BS4 documentation (https://www.crummy.com/software/BeautifulSoup/bs4/doc/#a-function). I'm under the impression that find_all() passes a bs4 tag element to the function and within the function you can do whatever you would do on a BS4 tag element. That seems not to be true. I get a string and obviously all my BS4 tag element operations raise an exception.

Two questions:

What exactly does find_all() pass to the function and how can the argument be used?
Besides a function how would otherwise be able to achieve the desired functionality?

What have you tried so far? Including a snippet of your html and other code to create a minimal, complete, and verifiable example will help us understand your specific situation. — BenG

wpercy wpercy · Accepted Answer · 2018-07-08T20:42:49

I'm under the impression that find_all() passes a bs4 tag element to the function and within the function you can do whatever you would do on a BS4 tag element.

Your impression is correct. From the bs4 documentation (emphasis mine):

... define a function that takes an element as its only argument.

So each element in the document is passed to the filter function, and if the function returns True for that element, it's included in the returned list of elements from .find_all(). This should really only be used as a last resort as it can be pretty performance intensive. I'm not entirely sure why you're getting strings, one possibility is that you're passing the function to the class_= param, which does pass in strings, but can't say without seeing your code.

I think what you should do is use .find_all() to get all <div>s with class "a", and then filter those to only include those that don't have class "b" using your own methods, something like this:

divs = soup.find_all('div', class_='a')
filtered_divs = [ div for div in divs if 'b' not in div.attrs['class'].split(' ') ]

But if you did want to use the filter function, it'd look something like

class_a_not_b(e):
    if 'class' not in e.attrs: return False
    return 'a' in e.attrs['class'].split(' ') and 'b' not in e.attrs['class'].split(' ')

divs = soup.find_all(class_a_not_b)

Filtering function issues in BS4 find_all() statement in Python

1 Answers