Implementation of Zero-Shot DeBERTa and IndoBERT for Aspect-Based Sentiment Analysis on Reviews of Five LLM Applications
DOI:
https://doi.org/10.31294/reputasi.v7i1.12790Keywords:
Aspect-Based Sentiment Analysis, Large Language Model, Zero-Shot Classification, IndoBERT, Google Play StoreAbstract
Large Language Model (LLM) applications such as ChatGPT, Gemini, Copilot, Claude, and Perplexity have been massively adopted in Indonesia, yet user experience evaluation remains largely limited to global sentiment analysis. This study implements Aspect-Based Sentiment Analysis (ABSA) using a dual-Transformer approach: DeBERTa zero-shot for aspect extraction and IndoBERT for sentiment classification on 5,000 Indonesian-language reviews from the Google Play Store across four aspect categories. Manual validation by two annotators on 300 samples yielded Cohen’s Kappa of (aspect) and (sentiment), both Moderate. Evaluation against the gold standard showed aspect accuracy of 37.5% (weighted F1 = 0.42) and sentiment accuracy of 64.7% (weighted F1 = 0.61). Sensitivity analysis across five hypothesis templates revealed inter-template Kappa of 0.19–0.63, confirming template selection impact on predictions. Comparative analysis reveals Copilot achieves the highest satisfaction (mean score 4.67), while Claude records the most complaints (36.9% negative). This study contributes a validated comparative ABSA framework for Indonesian-language LLM applications
References
Afuan, L., Hidayat, N., Hamdani, H., Ismanto, H., Purnama, B. C., & Ramdhani, D. I. (2025). Optimizing BERT Models with Fine-Tuning for Indonesian Twitter Sentiment Analysis. Journal of Wireless Mobile Networks, Ubiquitous Computing, and Dependable Applications, 16(2), 248–267. https://doi.org/10.58346/JOWUA.2025.I2.016
Al-Dossari, H. Z., & Altalasi, M. (2025). Multi-Aspect Sentiment Analysis of Arabic Café Reviews Using Machine and Deep Learning Approaches. Mathematics, 13(24), 3895. https://doi.org/10.3390/math13243895
Aravind, S. S., Rohilla, M., Kumar, O., Raj, S., & Sonal, D. (2025). Comparative study of BERT vs. traditional machine learning models for sentiment analysis. AIP Conference Proceedings, 3343(1). https://doi.org/10.1063/5.0292610
Bilal, A., Mirza, H. T., Khan, A. S., Ahmad, A., Hussain, I., & Ali, S. Z. (2025). Who dominates generative AI? Analyzing user feedback to identify common use cases and areas for improvement in ChatGPT, Copilot and Gemini. Knowledge and Information Systems, 67(11), 10797–10831. https://doi.org/10.1007/s10115-025-02550-y
Bustamin, A., Prayogi, A. A., Siswanto, D., Rafrin, M., & Nurdin, A. (2025). Text normalization for Indonesian slang words in sentiment analysis development. ICIC Express Letters, Part B: Applications, 16(2), 121–129. https://doi.org/10.24507/icicelb.16.02.121
Castilani, L. A., & Tuga, M. (2025). Customer Sentiment Analysis on OTA Platforms: Insights for Enhanced User Experience and Service Optimization. Proceedings of the 4th International Conference on Electronics Representation and Algorithm, 316–321. https://doi.org/10.1109/ICERA66156.2025.11087277
Chattu, K., Reddy, K. A. N., Veesam, S. B., Chirumamilla, P. S., Babu, V. D., Prakash, K., Bansal, S. K., Faruque, M. R. I., & Al-Mugren, K. S. (2025). Sentiment classification for Telugu using transformed based approaches on a multi-domain dataset. Scientific Reports, 15(1), 8124. https://doi.org/10.1038/s41598-025-05703-9
Chaudhary, M., Jain, C., & Anish, P. R. (2025). Exploring Zero-Shot App Review Classification with ChatGPT: Challenges and Potential. Proceedings of the 29th International Conference on Evaluation and Assessment in Software Engineering, 672–677. https://doi.org/10.1145/3756681.3757036
Harumy, T. H. F., Pauzi, & Arian. (2024). Sentiment Analysis of Halodoc Application Reviews Based on Service Quality Aspects Using BERT. Lecture Notes in Networks and Systems, 1089, 252–259. https://doi.org/10.1007/978-3-031-67195-1_30
Hashmi, E., & Yildirim, Ş. (2025). A robust hybrid approach with product context-aware learning and explainable AI for sentiment analysis in Amazon user reviews. Electronic Commerce Research, 25(6), 5139–5171. https://doi.org/10.1007/s10660-024-09896-5
Hidayatullah, A. F., Apong, R. A. A. H. M., Lai, D. T. C., & Qazi, A. (2025). Pre-trained language model for code-mixed text in Indonesian, Javanese, and English using transformer. Social Network Analysis and Mining, 15(1), 89. https://doi.org/10.1007/s13278-025-01444-9
Hossain, M. S. (2025). Emotional drivers of sustainable AI adoption: A sentiment analysis of early user feedback on the DeepSeek app. Sustainable Futures, 10, 100947. https://doi.org/10.1016/j.sftr.2025.100947
Jian, Z., Li, J., Wang, M., Yao, J., & Wu, Q. (2025). Aspect sentiment learning for Aspect-Level Sentiment Classification. Neural Networks, 191, 107758. https://doi.org/10.1016/j.neunet.2025.107758
Lashyn, Y., Trofymchuk, O. M., Zabolotnyi, S., Voitko, O., & Seabra, E. A. R. (2025). Sentiment analysis of texts using recurrent neural networks of the transformer architecture. Advanced Information Systems, 9(3), 91–101. https://doi.org/10.20998/2522-9052.2025.3.11
Liu, H., Xiong, K., Wu, S., Cao, P., Cheng, K., & Liu, X. (2025). Integrating multiple syntactic structures for enhanced aspect-based sentiment analysis. Engineering Applications of Artificial Intelligence, 158, 111297. https://doi.org/10.1016/j.engappai.2025.111297
Navaratna, A. R., & Saxena, D. K. (2025). Digital Governance Through Self-Regulation: A user-developer perspective of AI chatbots. Journal of Telecommunications and the Digital Economy, 13(3), 1–29. https://doi.org/10.18080/jtde.v13n3.1077
Prado-Sánchez, V. P., Domínguez-Díaz, A., de-Marcos, L. O., & Martínez-Herráiz, J. J. (2025). Zero-Shot Classification of Illicit Dark Web Content with Commercial LLMs: A Comparative Study on Accuracy, Human Consistency, and Inter-Model Agreement. Electronics, 14(20), 4101. https://doi.org/10.3390/electronics14204101
Riccosan, & Saputra, K. E. (2025). Multilabel classification sentiment analysis on Indonesian mobile app reviews. IAES International Journal of Artificial Intelligence, 14(5), 4226–4234. https://doi.org/10.11591/ijai.v14.i5.pp4226-4234
Setiawan, I. H., Rahardi, M., Aminuddin, A., & Abdulloh, F. F. (2024). Sentiment Analysis of Tokopedia Application Reviews on Google Play Store Using BERT. 2024 International Conference on Information Technology Systems and Innovation, 242–247. https://doi.org/10.1109/ICITSI65188.2024.10929357
Shaw, C., LaCasse, P. M., & Champagne, L. E. (2025). Exploring emotion classification of Indonesian tweets using large scale transfer learning via IndoBERT. Social Network Analysis and Mining, 15(1), 67. https://doi.org/10.1007/s13278-025-01439-6
Sinaga, F. M., Pangaribuan, J. J., Kelvin, Ferawaty, & Widjaja, A. E. (2025). Dynamic Sentiment Analysis on the Emergence of Pre-Trained Generative Model-Based Applications in Indonesia. International Journal of Advanced Computer Science and Applications, 16(12), 1150–1161. https://doi.org/10.14569/IJACSA.2025.01612111
Situmeang, S. I. G., Tambunan, S. R., Jevania, Simanjuntak, M. F., & Sinaga, S. (2025). Transformer and text augmentation for tourism aspect-based sentiment analysis. IAES International Journal of Artificial Intelligence, 14(6), 4614–4622. https://doi.org/10.11591/ijai.v14.i6.pp4614-4622
Srianan, S., Nanthaamornphong, A., & Phucharoen, C. (2025). Advancing tourism sentiment analysis: a comparative evaluation of traditional machine learning, deep learning, and transformer models on imbalanced datasets. Information Technology and Tourism, 27(4), 1011–1045. https://doi.org/10.1007/s40558-025-00336-0
Walji, K., Erraissi, A., Zakrani, A., & Banane, M. (2025). From Review to Practice: A Comparative Study and Decision-Support Framework for Sentiment Classification Models. International Journal of Advanced Computer Science and Applications, 16(9), 699–709. https://doi.org/10.14569/IJACSA.2025.0160967
Zaid, S., Alharbi, A., & Samra, H. El. (2025). Multi-Aspect Sentiment Classification of Arabic Tourism Reviews Using BERT and Classical Machine Learning. Data, 10(11), 168. https://doi.org/10.3390/data10110168
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Fabriyan Fandi Dwi Imaniawan, Ragil Wijianto, Vadlya Maarif, Joko Dwi Mulyanto, Mustofa, Aprih Widayanto
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
