python - Beautifulsoup: parsing html – get part of href

Question

I'm trying to parse

<td height="16" class="listtable_1"><a href="http://steamcommunity.com/profiles/76561198134729239" target="_blank">76561198134729239</a></td>

for the 76561198134729239. and I can't figure out how to do it. what I tried:

import requests
from lxml import html
from bs4 import BeautifulSoup
r = requests.get("http://ppm.rep.tf/index.php?p=banlist&page=154")
content = r.content
soup = BeautifulSoup(content, "html.parser")
element = soup.find("td", 
{
    "class":"listtable_1",
    "target":"_blank"
})
print(element.text)

Answer 1

There are many such entries in that HTML. To get all of them you could use the following:

import requests
from lxml import html
from bs4 import BeautifulSoup

r = requests.get("http://ppm.rep.tf/index.php?p=banlist&page=154")
soup = BeautifulSoup(r.content, "html.parser")

for td in soup.findAll("td", class_="listtable_1"):
    for a in td.findAll("a", href=True, target="_blank"):
        print(a.text)

This would then return:

76561198143466239
76561198094114508
76561198053422590
76561198066478249
76561198107353289
76561198043513442
76561198128253254
76561198134729239
76561198003749039
76561198091968935
76561198071376804
76561198068375438
76561198039625269
76561198135115106
76561198096243060
76561198067255227
76561198036439360
76561198026089333
76561198126749681
76561198008927797
76561198091421170
76561198122328638
76561198104586244
76561198056032796
76561198059683068
76561197995961306
76561198102013044

Answer 2

"target":"_blank" is a class of anchor tag a within the td tag. It's not a class of td tag.

You can get it like so:

from bs4 import BeautifulSoup

html="""
<td height="16" class="listtable_1">
    <a href="http://steamcommunity.com/profiles/76561198134729239" target="_blank">
        76561198134729239
    </a>
</td>"""

soup = BeautifulSoup(html, 'html.parser')

print(soup.find('td', {'class': "listtable_1"}).find('a', {"target":"_blank"}).text)

Output:

76561198134729239

Answer 3

As others mentioned you are trying to check attributes of different elements in a single find(). Instead, you can chain find() calls as MYGz suggested, or use a single CSS selector:

soup.select_one("td.listtable_1 a[target=_blank]").get_text()

If, you need to locate multiple elements this way, use select():

for elm in soup.select("td.listtable_1 a[target=_blank]"):
    print(elm.get_text())

Answer 4

"class":"listtable_1" belong to td tag and target="_blank" belong to a tag, you should not use them together.

you should use Steam Community as an anchor to find the numbers after it.

OR use URL, The URL contain the info you need and it's easy to find, you can find the URL and split it by /:

for a in soup.find_all('a', href=re.compile(r'steamcommunity')):
    num = a['href'].split('/')[-1]
    print(num)

Code:

import requests
from lxml import html
from bs4 import BeautifulSoup
r = requests.get("http://ppm.rep.tf/index.php?p=banlist&page=154")
content = r.content
soup = BeautifulSoup(content, "html.parser")
for td in soup.find_all('td', string="Steam Community"):
    num = td.find_next_sibling('td').text
    print(num)

out:

76561198143466239
76561198094114508
76561198053422590
76561198066478249
76561198107353289
76561198043513442
76561198128253254
76561198134729239
76561198003749039
76561198091968935
76561198071376804
76561198068375438
76561198039625269
76561198135115106
76561198096243060
76561198067255227
76561198036439360
76561198026089333
76561198126749681
76561198008927797
76561198091421170
76561198122328638
76561198104586244
76561198056032796
76561198059683068
76561197995961306
76561198102013044

Answer 5

You could chain together two finds in gazpacho to solve this problem:

from gazpacho import Soup

html = """<td height="16" class="listtable_1"><a href="http://steamcommunity.com/profiles/76561198134729239" target="_blank">76561198134729239</a></td>"""
soup = Soup(html)
soup.find("td", {"class": "listtable_1"}).find("a", {"target": "_blank"}).text

This outputs:

'76561198134729239'