Beautiful Soup - Extracting info

0
votes

I'm having some issues trying to extract info from this html excerpt.

So far I'm using this to extract the html below. 

#//////////////////////////////
with open('soup.html','r') as f:

    soup = BeautifulSoup(f, 'html.parser')

base = soup.find_all('script', type="application/ld+json")


print(base)
#//////////////////////////////
  1. How do I pull out the URL for each line?
  2. how do I pull out the name for each line?

This is what I get:

[<script type="application/ld+json">
      {"@context":"http://schema.org","@type":"Organization","name":"Redfin","logo":"https://ssl.cdn-redfin.com/static-images/images/redfin-logo-transparent-bg-260x66.png","url":"https://www.redfin.com"}
</script>,
<script type="application/ld+json">
    [{"@context":"http://schema.org","name":"7316 Green St, New Orleans, LA 70118","url":"/LA/New-Orleans/7316-Green-St-70118/home/79443425","address":{"@type":"PostalAddress","streetAddress":"7316 Green St","addressLocality":"New Orleans","addressRegion":"LA","postalCode":"70118","addressCountry":"US"},"numberOfRooms":"6","@type":"SingleFamilyResidence"},{"@context":"http://schema.org","@type":"Product","name":"7316 Green St, New Orleans, LA 70118","offers":{"@type":"Offer","price":"624900","priceCurrency":"USD"},"url":"/LA/New-Orleans/7316-Green-St-70118/home/79443425"}]
</script>,
<script type="application/ld+json">
    [{"@context":"http://schema.org","name":"257 Cherokee St #2, New Orleans, LA 70118","url":"/LA/New-Orleans/257-Cherokee-St-70118/unit-2/home/144766248","address":{"@type":"PostalAddress","streetAddress":"257 Cherokee St #2","addressLocality":"New Orleans","addressRegion":"LA","postalCode":"70118","addressCountry":"US"},"numberOfRooms":"2","@type":"SingleFamilyResidence"},{"@context":"http://schema.org","@type":"Product","name":"257 Cherokee St #2, New Orleans, LA 70118","offers":{"@type":"Offer","price":"129500","priceCurrency":"USD"},"url":"/LA/New-Orleans/257-Cherokee-St-70118/unit-2/home/144766248"}]
</script>, <script type="application/ld+json">
2
beautifulsoup

2 Answers

0
votes

What you show as the result is a list of dicts, you should iterate it and get the values needed.

0
votes

Use json to read in the dictionary/json format and then you can call the item by the key name:

you will need to add:

import json

Then you can do:

#//////////////////////////////
with open('soup.html','r') as f:

    soup = BeautifulSoup(f, 'html.parser')

base = soup.find_all('script', type="application/ld+json")


for each in base:
    jsonData = json.loads(each.text)
    url = jsonData['url']
    name = jsonData['name']

    print ('Name: %s\nURL: %s\n' %(name, url))
#//////////////////////////////