Problem: My selenium scrapping script will not get me tweet ids, probably because of an issue with how I request them.
Details:
Hello everyone, I have a script found here (https://github.com/bpb27/twitter_scraping/blob/master/scrape.py) which goes into a twitter search and gets the ids of old tweets.
From this script I have changed two parts:
user = 'realdonaldtrump'
start = datetime.datetime(2010, 1, 1) # year, month, day
end = datetime.datetime(2016, 12, 7) # year, month, day
has become
user = 'metoo'
start = datetime.datetime(2017, 10, 24) # year, month, day
end = datetime.datetime(2017, 10, 25) # year, month, day
and
def form_url(since, until):
p1 = 'https://twitter.com/search?f=tweets&vertical=default&q=from%3A'
p2 = user + '%20since%3A' + since + '%20until%3A' + until + 'include%3Aretweets&src=typd'
return p1 + p2
has become
def form_url(since, until):
p1 = 'https://twitter.com/search?l=fr&q=%23'
p2 = user + '%20since%3A' + since + '%20until%3A' + until + 'include%3Aretweets&src=typd'
return p1 + p2
After my changes the script correctly goes to search and iterates over all tweets but does not grab any ids.
Here is the request part:
for tweet in found_tweets:
try:
id = tweet.find_element_by_name(id_selector).get_attribute('href').split('/')[-1]
ids.append(id)
except StaleElementReferenceException as e:
print('lost element reference', tweet)
Any ideas how to fix this?
