1
votes

i m having some problem while fetching the data from twitter for 10 companies , i am fetching 10 thousands records from twitter for each company and performing sentiment analysis on that but while fetching records it is getting stuck , and then it stops !i am fetching data performing sentiment analysis and creating seperate files and then merging into one file ,but code is getting stuck while fetching data for first company itself

import tweepy

from textblob import TextBlob

import pandas as pd

Access_token="`access token here`"
Access_token_secret="`access token secret here`"
Consumer_key="`consumer key here`"
Consumer_secret_key="`consumer secret key here`"

auth=tweepy.OAuthHandler(Consumer_key,Consumer_secret_key)
auth.set_access_token(Access_token,Access_token_secret)
api=tweepy.API(auth)

search_list = ['company names here in a list']

for n in search_list:

    positive_tweet = []

    negative_tweet = []

    neutral_tweet = []

    Final_list = []

    new_list = []
   tweets = tweepy.Cursor(api.search, q=str(n) + '-filter:retweets', lang='en', since='2018-01-01').items(10000)

    for tweet in tweets:

        print(tweet.text)

        analysis = TextBlob(tweet.text)

        if (analysis.sentiment.polarity) > 0:

            positive_tweet.append(analysis.sentiment.polarity)
        elif analysis.sentiment.polarity < 0:
            negative_tweet.append(analysis.sentiment.polarity)
        elif analysis.sentiment.polarity == 0:
            neutral_tweet.append(analysis.sentiment.polarity)
        else:
            print('Emotionless & No Opinion regarding tweet')
    total = len(positive_tweet) + len(negative_tweet) + len(neutral_tweet)
    percentage_of_postive_review = float(len(positive_tweet) / (total) * 100)
    new_list.append(n)
    new_list.append(percentage_of_postive_review)
    # print(new_list)
    if new_list != []:
        label = ['Company Name', 'Review']
        i = 0
        j = 2
        while i < j and j <= len(new_list):
            Final_list.append(new_list[i:j])
            i = i + 2
            j = j + 2
        df = pd.DataFrame.from_records(Final_list, columns=label)
        # print(df)
        df.to_csv('D:/Review_Rating/Company Review' + str(n) + '.csv', index=False)
BTW - I moved the first company to the end of the list, because it does 10,000 tweets and it created the rate-limit problem when it was at the beginning. - Life is complex