HATE SPEECH CLASSIFICATION IN LOW RESOURCE REGIONAL LANGUAGESUSING CROSS LINGUAL TRANSFER LEARNING AND LLM BASED DATA AUGMENTATION
DOI:
https://doi.org/10.54314/ma09f535Kata Kunci:
hate speech detection, low-resource languages, cross-lingual transfer, data augmentation, large language modelAbstrak
Abstract: Hate speech on social media in Indonesia is often written in regional languages, which automatic moderation systems largely ignore because annotated data are scarce. This study builds a new annotated corpus of 6,000 social media comments in Batak Toba (BT) and Melayu Deli (MD), labelled as hate speech, abusive, or neutral by native speakers with substantial agreement (Fleiss’ κ = 0.71 for BT and 0.74 for MD). Two pre-trained models, IndoBERT and XLM-RoBERTa (XLM-R), were evaluated in three scenarios: zero-shot cross-lingual transfer from an Indonesian hate speech dataset, few-shot fine-tuning, and full fine-tuning. A fourth scenario adds synthetic training data generated by a large language model (LLM) and passed through a three-stage filter covering deduplication, label consistency, and native-speaker auditing. XLM-R with full fine-tuning reached a macro-F1 of 76.1% (BT) and 80.9% (MD). Filtered LLM augmentation raised these to 79.4% and 83.1%, outperforming EDA and back-translation, whereas unfiltered synthetic data reduced performance on BT.
Keyword: hate speech detection; low-resource languages; cross-lingual transfer; data augmentation; large language model
Abstrak: Ujaran kebencian di media sosial Indonesia sering ditulis dalam bahasa daerah, yang sebagian besar diabaikan oleh sistem moderasi otomatis karena data beranotasi sangat terbatas. Penelitian ini membangun korpus beranotasi baru berisi 6.000 komentar media sosial dalam bahasa Batak Toba (BT) dan Melayu Deli (MD), yang dilabeli sebagai ujaran kebencian, kasar, atau netral oleh penutur asli dengan tingkat kesepakatan substansial (Fleiss’ κ = 0,71 untuk BT dan 0,74 untuk MD). Dua model pralatih, IndoBERT dan XLM-RoBERTa (XLM-R), dievaluasi dalam tiga skenario: transfer lintas bahasa zero-shot dari dataset ujaran kebencian berbahasa Indonesia, fine-tuning few-shot, dan fine-tuning penuh. Skenario keempat menambahkan data latih sintetis yang dihasilkan large language model (LLM) dan disaring melalui tiga tahap, yaitu deduplikasi, konsistensi label, dan audit penutur asli. XLM-R dengan fine-tuning penuh mencapai macro-F1 76,1% (BT) dan 80,9% (MD). Augmentasi LLM yang tersaring meningkatkannya menjadi 79,4% dan 83,1%, melampaui EDA dan back-translation, sedangkan data sintetis tanpa penyaringan justru menurunkan performa pada BT. Augmentasi tersaring juga menurunkan false positive rate pada komentar netral yang memuat istilah identitas etnis atau agama dari 18,4% menjadi 15,2% (BT)
Kata kunci: deteksi ujaran kebencian; bahasa sumber daya rendah; transfer lintas bahasa; augmentasi data; large language model
Referensi
Aji, A. F., Winata, G. I., Koto, F., Cahyawijaya, S., Romadhony, A., Mahendra, R., Kurniawan, K., Moeljadi, D., Prasojo, R. E., Baldwin, T., Lau, J. H., & Ruder, S. (2022). One country, 700+ languages: NLP challenges for underrepresented languages and dialects in Indonesia. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL), 7226–7249.
Alfina, I., Mulia, R., Fanany, M. I., & Ekanata, Y. (2017). Hate speech detection in the Indonesian language: A dataset and preliminary study. 2017 International Conference on Advanced Computer Science and Information Systems (ICACSIS), 233–238.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., et al. (2020). Language models are few-shot learners. Advances in Neural Information Processing Systems, 33, 1877–1901.
Cahyawijaya, S., Lovenia, H., Aji, A. F., Winata, G. I., Wilie, B., et al. (2023). NusaCrowd: Open source initiative for Indonesian NLP resources. Findings of the Association for Computational Linguistics: ACL 2023, 13745–13818.
Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., & Stoyanov, V. (2020). Unsupervised cross-lingual representation learning at scale. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), 8440–8451.
Davidson, T., Warmsley, D., Macy, M., & Weber, I. (2017). Automated hate speech detection and the problem of offensive language. Proceedings of the International AAAI Conference on Web and Social Media (ICWSM), 512–515.
Feng, S. Y., Gangal, V., Wei, J., Chandar, S., Vosoughi, S., Mitamura, T., & Hovy, E. (2021). A survey of data augmentation approaches for NLP. Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, 968–988.
Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5), 378–382.
Fortuna, P., & Nunes, S. (2018). A survey on automatic detection of hate speech in text. ACM Computing Surveys, 51(4), 1–30.
Hartvigsen, T., Gabriel, S., Palangi, H., Sap, M., Ray, D., & Kamar, E. (2022). ToxiGen: A large-scale machine-generated dataset for adversarial and implicit hate speech detection. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL), 3309–3326.
Ibrohim, M. O., & Budi, I. (2019). Multi-label hate speech and abusive language detection in Indonesian Twitter. Proceedings of the Third Workshop on Abusive Language Online, 46–57.
Koto, F., Rahimi, A., Lau, J. H., & Baldwin, T. (2020). IndoLEM and IndoBERT: A benchmark dataset and pre-trained language model for Indonesian NLP. Proceedings of the 28th International Conference on Computational Linguistics (COLING), 757–770.
Lauscher, A., Ravishankar, V., Vulić, I., & Glavaš, G. (2020). From zero to hero: On the limitations of zero-shot language transfer with multilingual transformers. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 4483–4499.
Loshchilov, I., & Hutter, F. (2019). Decoupled weight decay regularization. International Conference on Learning Representations (ICLR).
McNemar, Q. (1947). Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, 12(2), 153–157.
Pires, T., Schlinger, E., & Garrette, D. (2019). How multilingual is multilingual BERT? Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), 4996–5001.
Sap, M., Card, D., Gabriel, S., Choi, Y., & Smith, N. A. (2019). The risk of racial bias in hate speech detection. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), 1668–1678.
Sennrich, R., Haddow, B., & Birch, A. (2016). Improving neural machine translation models with monolingual data. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL), 86–96.
Wei, J., & Zou, K. (2019). EDA: Easy data augmentation techniques for boosting performance on text classification tasks. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 6382–6388.
Wilie, B., Vincentio, K., Winata, G. I., Cahyawijaya, S., Li, X., Lim, Z. Y., Soleman, S., Mahendra, R., Fung, P., Bahar, S., & Purwarianti, A. (2020). IndoNLU: Benchmark and resources for evaluating Indonesian natural language understanding. Proceedings of the 1st Conference of the Asia-Pacific Chapter of the ACL and the 10th International Joint Conference on NLP (AACL-IJCNLP), 843–857.
Winata, G. I., Aji, A. F., Cahyawijaya, S., Mahendra, R., Koto, F., Romadhony, A., et al. (2023). NusaX: Multilingual parallel sentiment dataset for 10 Indonesian local languages. Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics (EACL), 815–834.
Add 5–10 recent (2021–2026) references from SINTA/Scopus-indexed journals on Indonesian hate speech detection and regional-language NLP.
Unduhan
Diterbitkan
Terbitan
Bagian
Lisensi
Hak Cipta (c) 2026 Vira Aditya, Yunita Awi
Artikel ini berlisensiCreative Commons Attribution-ShareAlike 4.0 International License.
