Scrape paragraph with href inside

Question

This is the html:

<p class="myParagraph">
  Lorem ipsum dolor sit amet, consectetur adipiscing elit. Vivamus vel justo
  <a href="http://google.it" class="small-link" target="_blank">
    <span class="tco-ellipsis"></span>
    <span class="invisible">https://</span>
    <span class="js-display-url">google.it</span>
    <span class="invisible">lpage/events/?ref=page_internal&amp;mt_nav=0&amp;locale2=it_IT</span>
    <span class="tco-ellipsis">
      <span class="invisible">&nbsp;</span>…
    </span>
  </a> ornare, suscipit nisl eget, aliquam augue. Aenean quis pretium
</p>

If i use tree.xpath('//p/text()') it only returns me

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Vivamus vel justo

instead of

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Vivamus vel justo ornare, suscipit nisl eget, aliquam augue. Aenean quis pretium

I also tried tree.xpath('string(//p)') here How can i take both complete paragraph and href? Not every time there's a a element inside paragraph

mzjn mzjn · Accepted Answer · 2019-08-29T09:17:20

xpath('//p/text()') returns a list of strings. Join these strings to get the wanted result.

from lxml import html

doc = """<p class="myParagraph">
  Lorem ipsum dolor sit amet, consectetur adipiscing elit. Vivamus vel justo
  <a href="http://google.it" class="small-link" target="_blank">
    <span class="tco-ellipsis"></span>
    <span class="invisible">https://</span>
    <span class="js-display-url">google.it</span>
    <span class="invisible">lpage/events/?ref=page_internal&amp;mt_nav=0&amp;locale2=it_IT</span>
    <span class="tco-ellipsis">
      <span class="invisible">&nbsp;</span>…
    </span>
  </a> ornare, suscipit nisl eget, aliquam augue. Aenean quis pretium
</p>"""

root = html.fromstring(doc)
print("".join([t for t in root.xpath("//p/text()")]))

Scrape paragraph with href inside

1 Answers