Sentiment Analysis: Harder Than It Looks

Sentiment analysis, despite its appealing name, proves to be a complex task rather than a straightforward solution. While the concept may seem simple, the challenges lie in the ambiguity of the labels, the structural nature of the phenomena, and the limited transferability of accuracy from public datasets to real-world applications.

There are three distinct tasks within sentiment analysis, each with its own unique requirements and limitations. Document polarity involves determining whether a review is positive or negative, a relatively straightforward task but not always the primary concern for businesses. Aspect-based sentiment, on the other hand, focuses on identifying specific aspects mentioned in the text, such as battery, shipping, or price, and analyzing the sentiment towards each aspect separately.

This task is often what product teams require, and it differs from document polarity as the answer is a set of pairs rather than a single label.

Emotion or intent analysis is another layer added to sentiment analysis, where the goal is to determine if the writer is expressing anger, identifying potential churn risk, or indicating the need for escalation. However, these categories are inherently tied to business context and lack standardized public datasets, making them more challenging to address effectively. These tasks should be treated as ordinary classification problems with custom labels specific to the business requirements.

The accuracy of sentiment analysis tools is often misleading due to the underspecified nature of the labels and the structural complexities involved. Negation, for example, can significantly impact the sentiment of a sentence. A bag-of-words model struggles to handle negation correctly, as in the case of "room was not clean," where the presence of "not" reverses the sentiment of "clean."

This issue is further exacerbated by the common practice of removing stop words, which eliminates the negation marker entirely. To address this, various fixes can be implemented, ranging from simple bigrams (like Wang and Manning's Baselines and Bigrams) to more advanced contextual models using transformer architectures.

Another important factor to consider is sarcasm, which poses a significant challenge in sentiment analysis. The inability of models to detect sarcasm is often attributed to a modeling limitation, but research suggests that human annotators also struggle to identify sarcasm from text alone. They often require additional context before making a judgment, and it is this context that classifiers frequently fail to provide.

This highlights the importance of considering the surrounding context when evaluating sentiment analysis tools, especially in high-volume environments such as social media.

Domain shift is another crucial consideration in sentiment analysis. The vocabulary used to express sentiment can vary significantly across different domains, leading to misinterpretations if a model is trained on data from one domain and applied to another. For example, "unpredictable" may be considered praise for a plot but a defect in a car, while "small" may be positive for a phone but negative for a hotel room.

These domain-specific differences are well-documented in the sentiment domain adaptation literature, emphasizing the need for careful consideration when selecting a model or API for sentiment analysis.

To ensure accurate sentiment analysis, it is essential to define the labels clearly and conduct rigorous evaluation. Skewing the evaluation set towards hard cases and computing inter-annotator agreement before evaluating any model can provide a more realistic assessment of performance. Accuracy alone is often misleading, particularly when dealing with imbalanced datasets.

Instead, focusing on per-class recall can be more informative, as the class of interest is typically the rarest one. Additionally, thoroughly examining errors and addressing definitional disagreements can significantly improve the accuracy of sentiment analysis models.