PERBANDINGAN KESESUAIAN PENDAPAT MANUSIA DAN AI (CHATGPT, GEMINI, DEEPSEEK) MENGGUNAKAN PENDEKATAN GROUND TRUTH

Authors

  • Kelvin Kelvin Universitas Mikroskil
  • Sunaryo Winardi Universitas Mikroskil
  • Handoko Handoko Universitas Mikroskil
  • Erwin Setiawan Panjaitan Universitas Mikroskil
  • Rivaldi Lubis Universitas Mikroskil

DOI:

https://doi.org/10.54314/jssr.v9i3.6784

Keywords:

Large Language Models, Sentiment Analysis, Ground Truth, Cohen's Kappa, Spearman's Rank Correlation

Abstract

Abstract: The rapid advancement of Large Language Models (LLMs) has significantly improved sentiment analysis capabilities. However, the extent to which these models produce sentiment classifications consistent with human judgment remains an important research question. This study aims to evaluate and compare the agreement of ChatGPT, Gemini, and DeepSeek with human-generated ground truth in sentiment analysis. A total of 1,497 product reviews were collected from the Sephora Products and Skincare Reviews dataset. Three independent annotators labeled each review as positive, neutral, or negative to establish the ground truth. The annotation reliability achieved a Fleiss' Kappa coefficient of 0.7053, indicating substantial agreement and confirming the reliability of the ground truth for evaluation purposes. Subsequently, the three LLMs performed sentiment classification using an Expectation–Role–Action (ERA) prompting strategy. Model performance was assessed using Cohen's Kappa to measure agreement with the ground truth and Spearman's rank correlation to evaluate the consistency of sentiment rankings. The results show that DeepSeek achieved the highest performance, with an average Cohen's Kappa of 0.753 and an average Spearman's correlation coefficient of 0.860, followed by Gemini (κ = 0.662; ρ = 0.834), while ChatGPT demonstrated the lowest agreement (κ = 0.223; ρ = 0.367). These findings indicate that the three LLMs exhibit significantly different levels of agreement with human judgment, with DeepSeek producing sentiment classifications that most closely align with the established ground truth.

 Keywords: Large Language Models, Sentiment Analysis, Ground Truth, Cohen's Kappa, Spearman's Rank Correlation.

 

Abstrak: Perkembangan Large Language Models (LLMs) telah meningkatkan kemampuan analisis sentimen berbasis kecerdasan buatan. Namun, tingkat kesesuaian hasil klasifikasi sentimen yang dihasilkan oleh berbagai LLM terhadap penilaian manusia masih memerlukan evaluasi yang komprehensif. Penelitian ini bertujuan membandingkan tingkat kesesuaian hasil analisis sentimen ChatGPT, Gemini, dan DeepSeek terhadap ground truth yang diperoleh melalui anotasi manusia. Penelitian menggunakan 1.497 ulasan produk dari dataset Sephora Products and Skincare Reviews. Sebanyak tiga anotator independen melakukan pelabelan sentimen ke dalam kategori positif, netral, dan negatif untuk membentuk ground truth. Hasil pengujian reliabilitas menunjukkan nilai Fleiss' Kappa sebesar 0,7053, yang mengindikasikan tingkat kesepakatan Substantial Agreement, sehingga ground truth layak digunakan sebagai acuan evaluasi. Selanjutnya, ketiga model LLM melakukan klasifikasi sentimen menggunakan prompt berbasis Expectation–Role–Action (ERA). Tingkat kesesuaian hasil klasifikasi dievaluasi menggunakan Cohen's Kappa, sedangkan konsistensi hubungan dengan ground truth dianalisis menggunakan korelasi Spearman. Hasil penelitian menunjukkan bahwa DeepSeek memberikan performa terbaik dengan rata-rata Cohen's Kappa sebesar 0,753 dan rata-rata korelasi Spearman sebesar 0,860, diikuti oleh Gemini (κ = 0,662; ρ = 0,834), sedangkan ChatGPT memperoleh tingkat kesesuaian terendah (κ = 0,223; ρ = 0,367). Temuan ini menunjukkan bahwa terdapat perbedaan kemampuan interpretasi sentimen antar model LLM, dengan DeepSeek menghasilkan klasifikasi yang paling mendekati penilaian manusia pada dataset yang digunakan.

Kata kunci: Large Language Model, Analisis Sentimen, Ground Truth, Cohen's Kappa, Spearman Correlation.

Downloads

Download data is not yet available.

References

O. Campesato, Large Language Models: An Introduction. Germany: De Gruyter, 2024.

M. Ren, "Advancements and Applica-tions of Large Language Models in Natural Language Processing: A Comprehensive Review," Applied and Computational Engineering, vol. 97, pp. 55–63, 2024. doi : 10.54254/2755-2721/97/20241406

T. R. Feyijimi, J. O. Aliu, A. E. Oke, and D. O. Aghimien, “ChatGPT’s expanding horizons and trans-formative impact across domains: A critical review of capabilities, challenges, and future directions,” Computers, vol. 14, no. 9, p. 366, 2025, doi: 10.3390/computers14090366

Z. Chkirbene, R. Hamila, A. Gouissem, and U. Devrim, "Large Language Models (LLM) in Industry: A Sur-vey of Applications, Challenges, and Trends," in 2024 IEEE 21st In-ternational Conference on Smart Communities: Improving Quality of Life using AI, Robotics and IoT (HONET), Doha, Qatar, 2024, pp. 229-234, doi: 10.1109/HONET63146.2024.10822885.

H. Huang, A. A. Zavareh and M. B. Mustafa, "Sentiment Analysis in E-Commerce Platforms: A Review of Current Techniques and Future Di-rections," IEEE Access, vol. 11, pp. 90367-90382, 2023, doi: 10.1109/ACCESS.2023.3307308.

R. Sharma, B. Singh, and A. Khampar-ia, “Machine learning and genera-tive AI techniques for sentiment analysis with applications,” in Ma-chine Learning and Generative AI Techniques for Sentiment Analysis with Applications, pp. 183–208, 2025. doi: 10.1002/9781394280735.ch10

S. Ivanov, K. Volchek, and C. Brito, “Generative AI for sentiment anal-ysis,” in Information and Commu-nication Technologies in Tourism 2025. ENTER 2025, L. Nixon, A. Tuomi, and P. O’Connor, Eds. Cham: Springer, 2025, pp. 129 - 139. doi: 10.1007/978-3-031-83705-0_11

Z. Ji, N. Lee, R. Frieske, T. Yu, D. Su, Y. Xu, E. Ishii, Y. J. Bang, A. Ma-dotto, and P. Fung, “Survey of hal-lucination in natural language gen-eration,” ACM Computing Sur-veys, vol. 55, no. 12, art. 248, pp. 1–38, Mar. 2023, doi: 10.1145/3571730.

J. Shuford, "Examining Ethical Aspects of AI: Addressing Bias and Equity in the Discipline," Journal of Arti-ficial Intelligence General Science (JAIGS), vol. 3, no. 1, pp. 262–280, 2024, doi: 10.60087/jaigs.v3i1.119.

A. Jadon, "Ethical AI Development: Mitigating Bias in Generative Mod-els," in Interplay of Artificial Gen-eral Intelligence with Quantum Computing: Towards Sustainabil-ity, C. K. K. Reddy, S. Joseph, H. Joshi, M. Ouaissa, and M. M. Hanafiah, Eds. Cham: Springer Na-ture Switzerland, 2025, pp. 123–136, doi: 10.1007/978-3-031-87931-9_10.

F. Cabitza, A. Campagner, and V. Ba-sile, "Toward a Perspectivist Turn in Ground Truthing for Predictive Computing," in Thirty-Seventh AAAI Conf. Artif. Intell., Thirty-Fifth Conf. on Innovative Applica-tions of AI, Thirteenth Symp. on Educational Advances in AI, B. Williams, Y. Chen, and J. Neville, Eds., Washington, DC, USA, Feb. 7–14, 2023, pp. 6860–6868, doi: 10.1609/aaai.v37i6.25840.

Aka, K. Burke, A. Bauerle, C. Greer, and M. Mitchell, "Measuring Mod-el Biases in the Absence of Ground Truth," in AIES '21: Proc. 2021 AAAI/ACM Conf. AI, Eth-ics, and Society, Virtual Event, USA, pp. 327–335, 2021, doi: 10.1145/3461702.3462557.

S. Vanbelle, C. Engelhart, and E. Blix, “A comprehensive guide to study the agreement and reliability of multi-

observer ordinal data,” BMC Medical Research Methodology, vol. 24, no. 310, 2024. [Online]. Available: https://doi.org/10.1186/s12874-024-02431-y.

W. Gan, Z. Qi, J. Wu, and J. C.-W. Lin, “Large language models in educa-tion: Vision and opportunities,” in Proc. IEEE Int. Conf. Big Data (BigData), Sorrento, Italy, 2023, pp. 4776–4785. doi: 10.1109/BigData59044.2023.10386291

Z. A. Nazi and W. Peng, “Large lan-guage models in healthcare and medical domain: A review,” In-formatics, vol. 11, no. 3, p. 57, 2024. doi: 10.3390/informatics11030057

R. Singh et al., "ChatGPT vs. Gemini: Comparative accuracy and effi-ciency in Lung-RADS score as-signment from radiology reports," Clinical Imaging, vol. 121, p. 110455, 2025. doi: 10.1016/j.clinimag.2025.110455

P. P. Nian et al., "ChatGPT and Google Gemini are Clinically Inadequate in Providing Recommendations on Management of Developmental Dysplasia of the Hip Compared to AAOS CPGs," Journal of the Pedi-atric Orthopaedic Society of North America, vol. 10, p. 100135, 2025. doi: 10.1016/j.jposna.2024.100135

Z. Zhan, S. Zhou, H. Zhou, J. Deng, Y. Hou, J. Yeung, and R. Zhang, "An evaluation of DeepSeek Models in Biomedical Natural Language Pro-cessing," arXiv preprint, arXiv:2503.00624, 2025. [Online]. Available: https://arxiv.org/abs/2503.00624

NadyinKy. Sephora products and skin-care reviews [Internet]. Kaggle; 2023 [cited 2025 Mar 12]. Availa-ble from: https://www.kaggle.com/datasets/nadyinky/sephora-products-and-skincare-reviews.

K. A. A. Al-Hameed, “Spearman's cor-relation coefficient in statistical analysis,” Int. J. Nonlinear Anal. Appl., vol. 13, no. 1, pp. 3249–3255, 2022, doi: 10.22075/ijnaa.2022.6079.

Y. Sun, D. Sheng, Z. Zhou et al., “AI hallucination: Towards a compre-hensive classification of distorted information in artificial intelli-gence-generated content,” Humani-ties and Social Sciences Communi-cations, vol. 11, art. 1278, 2024, doi: 10.1057/s41599-024-03811-x.

Permata, R., & Julianty, L. C. (2025). Towards an Automated Essay Evaluation System NLP Based Text Embeddings and Similarity Metrics. Digital Zone: Jurnal Teknologi Informasi dan Komu-nikasi, 16(1), 37–46. https://doi.org/10.31849/digitalzone.v16i1.26541.

D. C. Howell, “Chi-Square Test: Analy-sis of Contingency Tables,” in In-ternational Encyclopedia of Statis-tical Science, M. Lovric, Ed. Ber-lin, Heidelberg: Springer, 2025. doi: 10.1007/978-3-662-69359-9_109.

S. Vanbelle, C. Engelhart, and E. Blix, “A comprehensive guide to study the agreement and reliability of multi-observer ordinal data,” BMC Medical Research Methodology, vol. 24, no. 310, 2024. [Online]. Available: https://doi.org/10.1186/s12874-024-02431-y.

Downloads

Published

2026-06-30

Issue

Vol. 9 No. 3 (2026): June 2026

Section

Artikel

License

Copyright (c) 2026 Kelvin Kelvin, Sunaryo Winardi, Handoko Handoko, Erwin Setiawan Panjaitan, Rivaldi Lubis

Creative Commons License

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

How to Cite

PERBANDINGAN KESESUAIAN PENDAPAT MANUSIA DAN AI (CHATGPT, GEMINI, DEEPSEEK) MENGGUNAKAN PENDEKATAN GROUND TRUTH. (2026). JOURNAL OF SCIENCE AND SOCIAL RESEARCH, 9(3), 5181-5191. https://doi.org/10.54314/jssr.v9i3.6784