Issue with scraping tweets using python

Question

I am trying to scrape tweets from one webpage within a certain timeframe.

To do so I am using this link which only searches within the timeframe I have specified:

https://twitter.com/search?f=tweets&q=subwaydstats%20since%3A2016-08-22%20until%3A2018-08-22

This is my code:

import pandas as pd
import datetime as dt
import urllib.request
from bs4 import BeautifulSoup

url = 'https://twitter.com/search?f=tweets&q=subwaydstats%20since%3A2016-08-22%20until%3A2018-08-22'
thepage = urllib.request.urlopen(url)
soup = BeautifulSoup(driver.page_source,"html.parser")

i = 1
for tweet in soup.find_all('div', {'class': 'js-tweet-text-container'}):
    print(tweet.find('p', {'class': 'TweetTextSize'}).text.encode('UTF-8'))
    print(i)
    i += 1

The above code works when I am scraping from within the actual twitter page for the subwaystat user.

For this reason I don't understand why it doesn't work for the search page even though the html appears to be the same to me.

I am a total beginner so I'm sorry if this is a dumb question. Thank you!

Twitter does offer an API which you can use. Docs for Standard Search API: developer.twitter.com/en/docs/tweets/search/api-reference/… — Ben10
And there is a non-official Python wrapper to make requests to the api much easier: github.com/bear/python-twitter — Ben10
Hi, I have considered using the API but it only lets you get tweets from the last seven days. I suppose I could just have it continuously generating and store them somewhere but I was trying to look into alternate methods. — agra94
Scraping the Twitter website is against the Twitter Terks of Service and your IP address is likely to be blocked if this is detected. See twitter.com/tos — Andy Piper

Ben10 Ben10 · Accepted Answer · 2018-08-25T07:20:35

There is a Twitter API - Twitter Search API docs: https://developer.twitter.com/en/docs/tweets/search/api-reference/get-search-tweets which using a non-official Python wrapper: https://github.com/bear/python-twitter makes it super easy to get tweets.

However, if you want to scrape the HTML, then it's a lot more difficult. I was doing something similar - scraping an angular app, however, the actual HTML you see on the screen is actually rendered through "front-end javascript". Requests and urllib, only get the basic HTML but does not run the javascript.

You could use selenium which is basically a browser which you can automate task on. Since it behaves as a browser, it actually runs that front-end javascript, meaning you will be able to scrape the webpage.

A great article here explains the different ways you can scrape twitter https://medium.com/@dawran6/twitter-scraper-tutorial-with-python-requests-beautifulsoup-and-selenium-part-2-b38d849b07fe

Issue with scraping tweets using python

1 Answers