Real-time sentiment analysis of Twitter public stream

Abstract

Sentiment analysis on Twitter public stream has been a topic of research recently. Several non-commercial libraries and software were developed to perform sentiment analysis, however none of them performed the analytics in real-time for Twitter data. Performing the same task in real-time can gives us insight of Twitter users public opinions regarding recent happenings of the time that analysis was made. In this thesis work, we propose a full-stack architecture with a software prototype that performs real- time sentiment analysis on Twitter public stream. We address the problem using large- scale online learning and specifically online parallel decision trees. Large-scale learning is utilized due to the fact that social media website such as Twitter produce data with high volume (around 5800 tweets per second in 2014) and in addition, there is a high time constraint (up to seconds) in real-time analytics in both learning, processing and query response time. Moreover, Twitter stream data arrives instance-by-instance and therefore we have utilized online learning with incremental and per-instance learning flexibility. SAMOA is a framework that provides support for a set of scalable online learning algorithms such as Vertical Hoeffding Tree. We use SAMOA’s VHT learner with Apache Storm as our Stream Processing Engine. However, utilizing only VHT and Apache Storm cannot solve the problem at hand. Therefore, we also developed an open- source Java library called Sentinel that enables real-time Twitter stream reading, in- memory pre-processing computations and data structures, feature selection, frequent miner algorithms and etc. that completes our architecture. In Chapter 3, we show the architecture of our solution and its applicability and usefulness is shown in chapter 4.