XPath select until next tag

1
votes

I'm scraping https://en.wikipedia.org/wiki/Gadsden,_Alabama and I want to grab the "Geography and climate" paragraph and all the text() elements until the next h2 tag occurs (which is "Demographics").

I don't want to grab the table in here.

the XPath code I've tried so far is //span[@id='Geography_and_climate']/following::pwhich selects everything until the page ends.

I've tried answers' code of similar problems on StackOverflow but nothing works for me, maybe I'm not an expert to understand their code.

1
pythonxpathscrapyxpath-1.0
you may use loop to get next element and check if it is already h2 or not. - furas
@furas I don't know how to do that. - Haseeb
I was thinking about loop with item = item.findNextSibling() until item.name == 'h2' but it is code in BeautifulSoup - and if you use scrapy then it is not so useful. - furas

1 Answers

1
votes

Try this XPath to select all paragraphs after Geography&Climate and before Demographics

//p[preceding-sibling::h2[1][span[@id='Geography_and_climate']]]