IITM LOGO V17 FILE 1

IITM Journal of Information Technology

ISSN (P) 2395-5457 | Single Blind Peer Reviewed Journal

Published By
INSTITUTE OF INNOVATION IN TECHNOLOGY & MANAGEMENT
Affiliated to GGSIPU, NAAC Grade ‘A’, ISO 14001:2015, 17020:2012, 21001:2018 & 50001:2018 Certified,
A Grade by GNCTD, A++ Grade by SFRC

An Attention-Enhanced DeBERTa BiLSTM Hybrid Architecture for Robust Multi Class Sentiment Classification

Mansi Bansal1, Saurabh Sharma2, Sonal Singh3, Priyanka Singh4
1,4Assistant Professor, 2, 3 Research Scholar, 1Department of Computer Science, 3Department of Information Technology,
2,4School of Computer Science Engineering and Technology 1GGSIPU, Delhi, India, 2Bennett University, Greater Noida, Uttar Pradesh, India,
Government Engineering College, Bilaspur, Chhattisgarh, India, Rishihood University, Bahalgarh, Sonipat, Haryana, India 

Abstract: The widespread sharing of user-generated text on social media, review platforms, and online forums has intensified the demand for accurate, scalable, and context-aware sentiment analysis. Despite significant advancements in deep learning, sentiment classification remains challenging due to contextual ambiguity, linguistic diversity, domain dependency, long-distance semantic relationships, and class imbalance in real-world datasets. Traditional machine learning approaches rely heavily on handcrafted features and shallow representations, which limit their ability to capture deep semantic context. Although transformer-based models provide strong contextual embeddings, standalone transformers may not fully exploit sequential temporal dependencies that are essential for fine-grained sentiment modelling. In this work, we present an enhanced hybrid deep learning framework that extends existing transformer recurrent architectures by integrating a pre-trained DeBERTa-v3-base encoder with a stacked BiLSTM network and an explicit additive attention mechanism for multi-class sentiment classification. The DeBERTa encoder leverages disentangled attention and large-scale pretraining to generate rich contextualized representations, effectively modelling complex semantic relationships. The BiLSTM layer further refines these embeddings by capturing bidirectional sequential dependencies, while the attention layer selectively emphasizes sentiment discriminative tokens within the text. To address dataset imbalance and improve lexical diversity, word embedding-based data augmentation is employed. The proposed architecture is evaluated on widely used benchmark datasets including IMDb, Twitter US Airline, and Sentiment140, and is compared against classical machine learning models, standalone deep learning approaches, and transformer–recurrent hybrid baselines. Experimental results demonstrate consistent improvements in accuracy, precision, recall, and F1-score, particularly on short-form social media text. The proposed framework is scalable and generalizable, offering a robust solution for real-world applications in business intelligence, social analytics, and decision-support systems.

Keywords: Sentiment analysis, DeBERTa-v3-base, Bidirectional Long Short-Term Memory (BiLSTM), attention mechanism, Transformer, hybrid deep learning, text classification.

References:

  1. N. A. Semary, W. Ahmed, K. Amin, P. Pławiak, and M. Hammad, "Improving sentiment classification using a ROBERTa-based hybrid model," Frontiers in Human Neuroscience, vol. 17, p. 1292010, Dec. 2023.
  2. M. A. Jahin, M. S. H. Shovon, M. F. Mridha, M. R. Islam, and Y. Watanobe, "A hybrid transformer and attention-based recurrent neural network for robust and interpretable sentiment analysis of tweets," Scientific Reports, vol. 14, no. 24882, 2024.
  3. K. L. Tan, C. P. Lee, K. S. M. Anbananthen, and K. M. Lim, "RoBERTa-LSTM: A hybrid model for sentiment analysis with transformer and recurrent neural network," IEEE Access, vol. 10, pp. 21517- 21525,2022,doi:10.1109/ACCESS.2022.3152828.
  4. J. Devlin, M. W. Chang, K. Lee, and K. Toutanova, "BERT: Pre-training of deep bidirectional transformers for language understanding," in Proc. NAACL-HLT, 2019, pp. 4171-4186.
  5. A. Vaswani et al., "Attention is all you need," in Advances in Neural Information Processing Systems (NeurIPS), 2017, pp. 5998-6008.
  6. S. Hochreiter and J. Schmidhuber, "Long short-term memory," Neural Computation, vol. 9, no. 8, pp. 1735-1780, 1997.
  7. K. Cho et al., "Learning phrase representations using RNN encoder-decoder for statistical machine translation," in Proc. EMNLP, 2014.
  8. Y. Kim, "Convolutional neural networks for sentence classification," in Proc. EMNLP, 2014.
  9. T. Mikolov et al., "Efficient estimation of word representations in vector space," arXiv:1301.3781, 2013.
  10. J. Pennington, R. Socher, and C. Manning, "GloVe: Global vectors for word representation," in Proc. EMNLP, 2014.
  11. N. Reimers and I. Gurevych, "Sentence-BERT: Sentence embeddings using Siamese BERT-networks," in Proc. EMNLP-IJCNLP, 2019.
  12. H. Tan et al., "A hybrid RoBERTa-GRU model for sentiment analysis," Applied Sciences, 2023.
  13. H. Tan et al., "Hybrid RoBERTa-LSTM architecture for sentiment classification," IEEE Access, 2022.
  14. N. Umer, M. Imran, and S. Ullah, "Combining CNN and LSTM for sentiment analysis in social media data," IEEE Access, 2021.
  15. A. Rahat, S. Islam, and M. R. Islam, "Twitter US airline sentiment analysis using machine learning," Procedia Computer Science, 2019.
  16. M. Kumar et al., "Comparative study of machine learning techniques for sentiment analysis," Journal of Information Science, 2020.
  17. A. Goodrum et al., "Sentiment analysis in social media: Applications and challenges," IEEE Transactions on Computational Social Systems, 2020.
  18. M. Bansal et al., "Transformer-based multilingual sentiment analysis using XLM-ROBERTa," Expert Systems with Applications, 2023.
  19. S. Gupta et al., "Aspect-based sentiment analysis using hybrid CNN-RNN models," Knowledge-Based Systems, 2022.
  20. H. Basiri et al., "Sentiment analysis during COVID-19 using deep learning models," Information Processing & Management, 2021
Scroll to Top