Scrape Data From a Twitter Account and a Search Term

Image credit: Pixabay

Have you ever interested in what a science organization has recently discussed on Twitter? Have you ever been curious about how Twitter users have recently talked about a given topic? If you aim to answer those questions, this tutorial is for you.

Before answering the questions, you will need Python libraries and commands associated with data scraping and Twitter Application Programming Interface (API).

import tweepy
import pandas as pd
import time
consumer_key = "enter your consumer key"
consumer_key_secret = "enter your consumer key secret"
access_token = "enter your access token"
access_token_secret = "enter your acess token secret"
auth = tweepy.OAuthHandler(consumer_key, consumer_key_secret)
auth.set_access_token(access_token, access_token_secret)
api = tweepy.API(auth,wait_on_rate_limit=True)
tweets = []
def username_tweets_to_csv(username, recent):
try:
tweets = tweepy.Cursor(api.user_timeline,id=username).items(recent)
tweets_info = [[tweet.created_at, tweet.id, tweet.text] for tweet in tweets]
tweets_df = pd.DataFrame(tweets_info,columns=['Datetime', 'Tweet Id', 'Text'])
tweets_df.to_csv('{}-tweets.csv'.format(username), sep=',', index = False)
except BaseException as e:
print('failed on_status,',str(e))
time.sleep(3)
username = 'NSF'
recent = 150
username_tweets_to_csv(username, recent)
tweets = []
def text_query_to_csv(text_query,recent):
try:
tweets = tweepy.Cursor(api.search,q=text_query).items(recent)
tweets_list = [[tweet.created_at, tweet.id, tweet.text] for tweet in tweets]
tweets_df = pd.DataFrame(tweets_list,columns=['Datetime', 'Tweet Id', 'Text'])
tweets_df.to_csv('{}-tweets.csv'.format(text_query), sep=',', index = False)
except BaseException as e:
print('failed on_status,',str(e))
time.sleep(3)
text_query = 'climate change'
recent = 15
text_query_to_csv(text_query, recent)

Communication psychology researcher, data scientist.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store