Review on A Deep Learning for Sentiment Analysis from Twitter

Posted by Mohamad Ivan Fanany

Printed version

This writing summarizes and reviews a deep learning for sentiment analysis from twitter: Coooolll: A Deep Learning System for Twitter Sentiment Classification

Addressed problem:

  • Twitter sentiment classification within a supervised learning framework.
  • Generate features unsupervisedly using deep learning from 10M tweets collected by positive and negative emoticons, without any manual annotation.
  • Embed deep learning features with state-of-the-art features.
  • Classify sentiment using the features generated by deep learning and compared the result with using the joint features.

Previous works:

  • Twitter sentiment classification aims to classify the sentiment polarity of a tweet as positive, negative or neutral (Jiang et al., 2011; Hu et al., 2013; Dong et al., 2014).
  • The majority of existing approaches follow Pang et al. (2002) and employ machine learning algorithms to build classifiers from tweets with manually annotated sentiment polarity.
  • Under this direction, most studies focus on designing effective features to obtain better classification performance ( Pang and Lee, 2008; Liu, 2012; Feldman, 2013). For example:

Key ideas:

  • Coooolll is built in a supervised learning framework by concatenating the sentiment-specific word embedding (SSWE) features with the state-of-the-art hand-crafted features (STATE).
  • To obtain large-scale training corpora, train the SSWE from 10M tweets collected by positive and negative emoticons, without any manual annotation.
  • The proposed system can be easily re-implemented with the publicly available sentiment-specific word embedding.
  • Conduct experiments on both positive/negative/neutral and positive/negative classification of tweets.
  • Develop a deep learning for message-level Twitter sentiment classification.
  • Develop a neural network with hybrid loss function to learn SSWE, which encodes the sentiment information of tweets in the continuous representation of words.

Network architecture:

  • The proposed neural network for learning sentiment-specific word embedding is an extension of the traditional C&W model (Collobert et al., 2011).
  • Unlike C&W model that learns word embedding by only modeling syntactic contexts of words, the proposed SSWEu captures the sentiment information of sentences as well as the syntactic contexts of words.
  • Given an original (or corrupted) ngram and the sentiment polarity of a sentence as the input, SSWEu predicts a two-dimensional vector for each input ngram.

Dataset:

  • 10 million tweets from Twitter Sentiment Analysis Track in SemEval 2014

Validation:

  1. Learn sentiment-specific word embedding (SSWE) (Tang et al., 2014), which encodes the sentiment information of text into the continuous representation of words (Mikolov et al., 2013; Sun et al., 2014).
  2. Concatenate the SSWE features with the STATE (state-of-the-art hand-crafted) features (Mohammad et al., 2013),
  3. Train the sentiment classifier with the benchmark dataset from SemEval 2013 (Nakov et al., 2013). The classifier is LibLinear (Fan et al., 2008)
  4. Test the trained model using test sets of SemEval 2014.

Features:

  • SSWE Features:
    • Given an original (or corrupted) ngram and the sentiment polarity of a sentence as the input, SSWEu predicts a two-dimensional vector for each input ngram.
    • The two scalars stand for language model score and sentiment score of the input ngram.
    • The training objectives of SSWEu are
      1. The original ngram should obtain a higher language model score than the corrupted ngram,
      2. The sentiment score of original should be more consistent with the gold polarity annotation of sentence than corrupted.
    • The loss function of SSWEu is the linear combination of two hinge losses: the syntactic loss and the sentiment loss.
    • After finish learning SSWE, explore min, average and max convolutional layers (Collobert et al., 2011; Socher et al., 2011; Mitchell and Lapata, 2010), to obtain the tweet representation.
    • The result is the concatenation of vectors derived from different convolutional layers.
  • STATE Features:
    • Re-implement the state-of-the-art hand-crafted features (Mohammad et al., 2013) for Twitter sentiment classification.
    • The STATE features:
    • All Caps :The number of words with all characters in upper case.
    • Emoticons:The presence of positive (or negative) emoticons and whether the last unit of a segmentation is emoticon.
    • Elongated Units:The number of elongated words (with one character repeated more than two times), such as gooood.
    • Sentiment lexicon:Several sentiment lexicons to generate features:
      • the number of sentiment words,
      • the score of last sentiment words,
      • the total sentiment score and
      • the maximal sentiment score for each lexicon.
    • Negation: The number of individual negations within a tweet.
    • Puctuation: The number of contiguous sequences of dot, question mark and exclamation mark.
    • Cluster: The presence of words from each of the 1,000 clusters from the Twitter NLP tool.
    • Ngrams: The presence of word ngrams (1-4) and character ngrams (3-5).

Results:

  • Among the 45 submitted systems including the SemEval 2013 participants, the proposed system (Coooolll) is ranked 2nd on the Twitter2014 test set of SemEval 2014 Task 9.
  • The performance of only using SSWE as features is comparable to the stateof-the-art hand-crafted features which verifies the effectiveness of the sentiment-specific word embedding.

My Review:

  • The paper is a nice reading for it refers to many state of the art sentiment analysis system.
  • The core of the system is the learning sentiment-specific word embedding features using deep learning. The details of algorithm is written on this paper.
  • Deep learning was used only to generate features. If the features extraction and classification can be performed in a single pass, this might brings deeper insights on how the deep classifier will also fine tune the extracted features.
Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s