Similar to the last post of mine on Python and data analysis, I was so inspired by the DataCamp.com material that I took the script I previously had and reworked it to show Sentiment Classification.

What is Sentiment?

In Machine Learning lingo, Sentiment is a classification of “Positive” or “Negative” intent, based upon the choice of words or language used.

How is Sentiment Calculated?

Often it is done using a Machine Learning Framework, that you train with specific sampled data and results.  In my case, however, I used a pre-built sentiment analysis package that is part of the Textblob library.

If you’re using Anaconda, you can install Textblob via: https://anaconda.org/conda-forge/textblob

GeeksForGeeks Approach

While I wrote the script below, I did find Textblob from someone else’s work who was already doing Twitter Analysis… I’ll cite them for reference as well: https://www.geeksforgeeks.org/twitter-sentiment-analysis-using-python/

As luck would have it, they too did an approach of Twitter Sentiment Analysis.  Their choice was to use TextBlob for the Sentiment Analysis and I went ahead and did the same thing in my script below.  As it turns out TextBlob is a great place to start but the accuracy I think will improve with a training based framework (as mentioned later on.)

The Concept

The idea I was trying to work out, was to build off the previous script that graphed Twitter user Tweets about subjects.  Instead of getting a figure of counts of tweets about subject 1 vs. subject 2, I decided to try and get user sentiment.  How many of the Tweets are positive about a single subject?

For example, if the sentiment of a cryptocurrency (i.e. Bitcoin) is normally 65% positive, and then the next day there are 3 times as many Tweets and they average 95% positive, it could be indicative of a pump in the price… and conversely, if there is a large shift in Negative sentiment, it could foretell Fear in the market and a potential Sell-Off.

Libraries Used

For this work, I made use of the Tweepy, TextBlob and Datetime libraries.

Tweepy was used to interface and authenticate with Twitter.

TextBlob was used to get basic sentiment scores.

Datetime was used to filter the results of Twitter data to a 24hr timeframe.

Code

Below is the Python script that takes in a subject (i.e. “bitcoin”), queries Twitter and then iterates over the text of each tweet, performing a Sentiment Analysis score.  These scores are tallied up and then a percentage is calculated of positive or negative sentiment on the subject.

[pastacode lang=”python” manual=”import%20tweepy%2C%20datetime%0Afrom%20textblob%20import%20TextBlob%0A%0Aaccess_token%20%3D%20’XXXXXXXXXXXXXX’%0Aaccess_token_secret%20%3D%20’XXXXXXXXXXXXXXXX’%0Aconsumer_key%20%3D%20’XXXXXXXXXXXXXXXXXX’%0Aconsumer_secret%20%3D%20’XXXXXXXXXXXXXXXXXX’%0A%0Aauth%20%3D%20tweepy.OAuthHandler(consumer_key%2C%20consumer_secret)%0Aauth.set_access_token(access_token%2C%20access_token_secret)%0A%0Aapi%20%3D%20tweepy.API(auth)%0A%0A%5Bpos%2C%20neg%5D%20%3D%20%5B0%2C0%5D%0Asent%20%3D%20”%0Aresult%20%3D%200%0A%0Adef%20query_twitter(q%2C%20max_tweets%3D50)%3A%0A%20%20%20%20for%20tweet%20in%20tweepy.Cursor(api.search%2C%20q%3Dq).items(max_tweets)%3A%0A%20%20%20%20%20%20%20%20if%20(datetime.datetime.now()%20-%20tweet.created_at).days%20%3C%201%3A%0A%20%20%20%20%20%20%20%20%20%20%20%20sentiment(tweet.text)%0A%20%20%20%20print(%5Bpos%2C%20neg%5D)%0A%20%20%20%20print(%22Subject%3A%20%22%2B%20%22’%22%20%2B%20q%20%2B%20%22’%22%20%2B%20%22%20is%20%22%20%2B%20str(%22%7B0%3A.2f%7D%22.format(result))%20%2B%20%22%25%20%22%20%2B%20sent)%0A%0Adef%20sentiment(tweet)%3A%0A%20%20%20%20global%20pos%0A%20%20%20%20global%20neg%0A%20%20%20%20analysis%20%3D%20TextBlob(tweet)%0A%20%20%20%20if%20analysis.sentiment.polarity%20%3E0%3A%0A%20%20%20%20%20%20%20%20print(‘Positive%3A%20’%20%2B%20tweet%20%2B%20’%5Cn’)%0A%20%20%20%20%20%20%20%20pos%20%3D%20pos%20%2B%201%0A%20%20%20%20elif%20analysis.sentiment.polarity%20%3C%200%3A%0A%20%20%20%20%20%20%20%20print(‘Negative%3A%20’%20%2B%20tweet%20%2B%20’%5Cn’)%0A%20%20%20%20%20%20%20%20neg%20%3D%20neg%20%2B%201%20%20%20%20%20%20%20%20%0A%0A%20%20%20%20return%20sentiment_percent(pos%2C%20neg)%0A%0Adef%20sentiment_percent(pos%2C%20neg)%3A%0A%20%20%20%20global%20sent%0A%20%20%20%20global%20result%0A%20%20%20%20total%20%3D%20pos%20%2B%20neg%0A%20%20%20%20if%20pos%20%3E%20neg%3A%0A%20%20%20%20%20%20%20%20sent%20%3D%20%22Positive%22%0A%20%20%20%20%20%20%20%20result%20%3D%20pos%20%2F%20total%20*%20100%0A%20%20%20%20else%3A%0A%20%20%20%20%20%20%20%20sent%20%3D%20%22Negative%22%0A%20%20%20%20%20%20%20%20result%20%3D%20neg%20%2F%20total%20*%20100%0A%20%20%20%20return%20%7Bsent%3Aresult%7D%0A%0Aquery_twitter(‘trump’%2C%20500)” message=”” highlight=”” provider=”manual”/]

Result Example

The results of the code above will output something similar to this:

[pastacode lang=”markup” manual=”%0ANegative%3A%20RT%20%40DeanObeidallah%3A%20Thanks%20to%20Trump%20we%20are%20now%20seeing%20Trump%20supporters%20in%20essence%20calling%20the%20survivors%20of%20the%20school%20shooting%20%22Fake%20news.%22%E2%80%A6%0A%0ANegative%3A%20%40NancyPelosi%20%40SpeakerRyan%20%40SenateMajLdr%20Need%20to%20halt%20Mueller%2FDemocrat%20lame%20attempt%20to%20get%20rid%20of%20President%20Trump.%20F%E2%80%A6%20https%3A%2F%2Ft.co%2FWc64lsX9Q5%0A%0ANegative%3A%20%40timberjack2004%20%40CNNPolitics%20%23DotardTrump%20is%20a%20lying%20hypocrite!%20He%20signed%20the%20law%20allowing%20mentally%20ill%20people%20to%20p%E2%80%A6%20https%3A%2F%2Ft.co%2FU5n7kP15Nc%0A%0ANegative%3A%20RT%20%40AaronBBrown%3A%20Now%20comes%20the%20blaming%20of%20people%20with%20mental%20illness%2C%20and%20the%20push%20to%20throw%20all%20of%20them%20in%20institutions.%20Trump%20will%20never%20f%E2%80%A6%0A%0ANegative%3A%20RT%20%40bidonkules%3A%20%40joanwalsh%20Bernie%20can’t%20help%20himself.He%20gave%20interviews%20to%20RT%2C%20Russian%20propaganda%20outlet%2C%20denigrating%20Hillary%20back%20in%202014-%E2%80%A6%0A%0APositive%3A%20Trump%20is%20dying%20to%20tweet%20right%20now%2C%20watch%20what%20he%20says%20after%2C%20%22it%20was%20such%20a%20great%20meting%2C%20great%20people%2C%20great%20voice%E2%80%A6%20https%3A%2F%2Ft.co%2Fx2pTrxl8Hp%0A%0APositive%3A%20RT%20%40ToDropADime2%3A%20The%20%23TwitterLockOut%20was%20a%20step%20in%20the%20right%20direction%20to%20purge%20%23RussiaBots%20%0A%0ANext%3A%20%23FacebookLockOut%20or%0A%23boycottfacebook%20!%E2%80%A6%0A%0APositive%3A%20RT%20%40amazingatheist%3A%20I%20think%20it’s%20extremely%20cool%20that%20Trump%20is%20doing%20this%20big%20public%20discussion%20of%20school%20shootings%20and%20bringing%20all%20points%E2%80%A6%0A%0APositive%3A%20Media%20har%20hintat%20om%20att%20Trump%20varit%20positiv%20till%20f%C3%B6rbud%2Fregleringar.%20S%C3%A5%20kommer%20detta.%20Troll%20level%3A%20grand%20master%20https%3A%2F%2Ft.co%2FUcG16Q1SjG%0A%0ANegative%3A%20I’m%20so%20sick%20of%20so%20called%20Trump%20supporters%20coming%20on%20social%20media%2C%20or%20talk%20radio%2C%20saying%20what’s%20Trump%20doing%3F%20%20He’s%20t%E2%80%A6%20https%3A%2F%2Ft.co%2FZZxUuh4EKX%0A%0APositive%3A%20RT%20%40goldengateblond%3A%20Trump%20loved%20the%20terrible%20concealed%20carry%20idea.%20It%E2%80%99s%20the%20most%20animated%20he%E2%80%99s%20been%20the%20whole%20time.%20Guarantee%20this%20will%20be%E2%80%A6%0A%0APositive%3A%20RT%20%40ReutersPolitics%3A%20JUST%20IN%3A%20Trump%20calls%20for%20end%20of%20gun-free%20zones%20near%20schools%2C%20endorses%20idea%20of%20teachers%20and%20others%20in%20schools%20being%20arm%E2%80%A6%0A%0APositive%3A%20RT%20%40Mariotte67897%3A%20Melania%20Trump’s%20parents’%20immigration%20status%20could%20be%20thanks%20to%20’chain%20migration’%20https%3A%2F%2Ft.co%2F3Y7lite8Kg%0A%0ANegative%3A%20Trump%E2%80%99s%20solutions%20were%20always%20always%20going%20to%20be%20the%20worst%20solutions%20https%3A%2F%2Ft.co%2FBjNOjCSzdg%0A%0APositive%3A%20RT%20%40RVAwonk%3A%20Donald%20Trump%20just%20went%20off%20script%20on%20live%20TV%20and%20promised%20shooting%20survivors%3A%20%22We’ll%20be%20doing%20the%20background%20checks.%22%0A%0A%23Parkla%E2%80%A6%0A%0APositive%3A%20%40krassenstein%20Trump%20is%20in%20a%20gun%20talk%20sit%20down%20right%20now.%20I%20can%20already%20see%20him%20forgetting%20about%20the%20meeting%20and%20bas%E2%80%A6%20https%3A%2F%2Ft.co%2FObRQ5zlgyU%0A%0A%5B184%2C%20116%5D%0ASubject%3A%20’trump’%20is%2061.33%25%20Positive” message=”” highlight=”” provider=”manual”/]

In the above output, we can see a lot of strong Sentiment.  TextBlob is nice and easy to get such a sentiment calculation, but I noticed it often makes mistakes.  I’ve noticed in my experiments, that TextBlob Sentiment tends to overly mark “positive” on Tweets.  Sometimes these Tweets are obviously Negative.

This is perhaps due to the lack of training on the data sets.

 

Better Analysis

A better approach to Sentiment Analysis would be in data training.  TextBlob isn’t capable of this (as far as I know), and this would require a deeper dive into Machine Learning frameworks like NLTK.  There’s a working example of data training using NLTK at: https://www.kaggle.com/ngyptr/python-nltk-sentiment-analysis

Once I get more comfortable with Machine Learning and Data manipulation, I’ll retry this experiment with the NLTK libraries – training the system on positive and negative comments, and see how it categorizes tweets (either better, the same or worse) in comparison to TextBlob’s built-in Sentiment Analysis system.

#

Comments are closed

Archives
Categories