Python bootcamp project

Background

The goal of the project is to analyze the tweeting behavior of the 100 US Senators in the US Senate. Each senator has a Twitter account and puts out tweets reflecting their thought and communicating with the people in the state they represent.

(Note that these tweets were downloaded early in 2017, so they reflect tweeting just before President Obama stepped down and President Trump began his term.)

The goal of the project questions is to guide you through the steps of getting the data, processing and cleaning it, putting it in a format that makes it easier to analyze and then doing some basic analysis. The last few questions ask you to see if whether a senator mentions a president or presidential candidate depends on the party that the senator is part of. For example, do Democratic senators mention Barack Obama in their tweets more or less than Republican senators?

For background information only: The file project/fetch_senator_tweets.py downloads tweets using the Python twitter package to interact with Twitter’s API. You will only be able to run that code if you set up your own Twitter account and follow the instructions at the start of the file regarding filling in the authentication information (CONSUMER_KEY, CONSUMER_SECRET, etc.).

I’ve already run the code mentioned above and downloaded the data for you. The downloaded information on the senators’ twitter accounts is in project/senators-list.json in the Github repository, while the downloaded tweets are in timelines.json. timelines.json is too big to put in the Github repository. You can find it at http://www.stat.berkeley.edu/~paciorek/transfer/timelines.json. Note that there are only 200 tweets for each senator because of limits on how many tweets can be accessed in a given request.

Questions

Load the senators-list.json and timelines.json files into Python as objects called senators and timelines.
What type of datastructure is timelines? How many timelines are there? What does each timeline correspond to?
Make a list of the number of followers each senator has.
What is the screen name of the senator with the largest number of followers.
Make a list of lists where the outer list represents senators and the inner list contains the text of each senator’s tweets, and call it tweets.
Write a function, called remove_punct, that takes a word and returns the word with all punctuation characters removed, except for those that occur within a word.
Write a function that takes tweet and returns a cleaned up version of the tweet. Here is an example function to get you started:
```
def clean(tweet):
    cleaned_words = [word.lower() for word in tweet.split() if
                 'http' not in word and
                 word.isalpha() and
                 word != 'RT']
    return ' '.join(cleaned_words)

clean(tweets[0][0])
```
Note that the function I’ve provided is a bit buggy - it has some problems with some tweets. If your goal is to convert the tweet into a discrete set of words, what is going wrong here? Fix up and extend the example function.
Use the following file to create a list, called stopwords, that contains common english words: http://www.textfixer.com/resources/common-english-words.txt. Make sure to pull the data into Python by writing Python code to download and suck the data into Python.
Write a function, called tokenize, which takes a tweet, cleans it, and removes all punctuation and stopwords.
Create a list of lists, tweets_content, using your tokenize function.
Create a list, tokens, where all 200 of each senator’s tweets are made into a single string. Hint: this syntax might be useful: " ".join(my_list_of_strings).
Create a Pandas dataFrame with the following columns: senator name or handle, party of the senator, and number of times a prominent politician is mentioned in each senator’s tweets. You might count the number of ‘Obama’, ‘Trump’, or ‘Clinton’ references.

You can use this to create the party column (1=Republican, 0=Democratic):
```
party = np.array([1,1,1,1,1,1,0,0,1,0,1,1,0,0,0,1,1,1,0,0,0,1,1,1,1,0,1,0,
1,1,1,0,0,0,0,0,0,1,0,1,1,1,0,1,1,1,0,1,1,1,1,0,0,0,0,0,0,0,0,0,1,1,1,0,1,
1,0,0,0,1,1,0,0,0,0,0,1,1,1,1,1,1,1,1,1,0,1,1,0,0,1,0,1,0,0,1,1,1,0,1])
```
That should correspond to the ordering in timelines but of course it would be more robust to create a dataFrame that has user names and party as columns and merge that with the count information.
Use a Poisson GLM to assess the relationship between party and number of Obama/Trump/Clinton mentions. Does one party tend to mention Obama/Trump/Clinton more in their tweets? Can you deduce a pattern by considering the party of the senator and the party of Obama/Trump/Clinton?

Here’s some syntax to help you get started:
```
import statsmodels.api as sm
model = sm.GLM(endog = .,  exog = ., family = sm.families.Poisson())
model.fit()
```
Does the statistical result make sense in light of the number of total mentions of Obama/Trump/Clinton by Republicans and the number of total mentions by Democrats?
Use matplotlib to make histograms of the number of Obama mentions by senator, stratified by party.

Is this consistent with the results of your statistical analysis?