OPTIMASI MODEL KLASIFIKASI PESAN PENIPUAN DIGITAL BERBAHASA INDONESIA MENGGUNAKAN XGBOOST DENGAN BAYESIAN OPTIMIZATION DAN SMOTE
DOI:
https://doi.org/10.54314/jssr.v9i3.7212Keywords:
Digital Fraud, Text Classification, XGBoost, Bayesian Optimization, SMOTEAbstract
Abstract: Digital fraud through short messages has become a national problem in Indonesia; the Indonesia Anti-Scam Centre recorded 637,055 reports with losses of IDR 9.1 trillion up to July 2026. Scam messages are hard to filter with keywords because perpetrators keep changing wording and phone numbers, while official promotional messages share similar linguistic patterns. This study aims to optimize a three-class classifier (normal, fraud, promotion) for Indonesian messages by combining XGBoost, SMOTE, and Bayesian Optimization based on the Tree-structured Parzen Estimator (TPE), and to compare TPE with Grid Search and Random Search under an equal budget of 25 trials. Messages were normalized (URLs, phone numbers, amounts, and slang) and represented with combined word and character n-gram TF-IDF features. On 2,320 messages, XGBoost + SMOTE + TPE achieved a macro F1 of 0.9555, an accuracy of 96.34%, and a fraud-class recall of 0.9507, slightly higher than default XGBoost (F1 0.9531) and Grid Search (F1 0.9529), with a much lighter model that trained 3.2 times faster. Grid Search showed overfitting to the validation folds. The ablation study showed that SMOTE did not help under moderate class imbalance (F1 without SMOTE 0.9581), and Multinomial Naïve Bayes remained the best model (F1 0.9693). Most errors occurred between fraud and promotional messages.
Keywords: Digital Fraud, Text Classification, XGBoost, Bayesian Optimization, SMOTE
Abstrak: Penipuan digital melalui pesan singkat telah menjadi masalah nasional; Indonesia Anti-Scam Centre mencatat 637.055 laporan dengan kerugian Rp9,1 triliun hingga Juli 2026. Pesan penipuan sulit disaring dengan kata kunci karena pelaku terus mengubah variasi kata dan nomor telepon, sementara pesan promosi resmi memiliki pola bahasa yang mirip. Penelitian ini bertujuan mengoptimasi model klasifikasi tiga kelas (normal, penipuan, promo) untuk pesan berbahasa Indonesia dengan menggabungkan XGBoost, SMOTE, dan Bayesian Optimization berbasis Tree-structured Parzen Estimator (TPE), serta membandingkan TPE dengan Grid Search dan Random Search pada anggaran yang sama, yaitu 25 percobaan. Pesan dinormalisasi (tautan, nomor telepon, nominal, dan kata tidak baku) lalu direpresentasikan dengan fitur TF-IDF gabungan n-gram kata dan karakter. Pada 2.320 pesan, XGBoost + SMOTE + TPE mencapai F1-macro 0,9555, akurasi 96,34%, dan recall kelas penipuan 0,9507, sedikit lebih tinggi daripada XGBoost default (F1 0,9531) dan Grid Search (F1 0,9529), dengan model yang jauh lebih ringan dan waktu pelatihan 3,2 kali lebih cepat. Grid Search menunjukkan gejala overfitting terhadap fold validasi. Uji ablasi menunjukkan SMOTE tidak membantu pada ketidakseimbangan kelas moderat (F1 tanpa SMOTE 0,9581), dan Multinomial Naïve Bayes tetap menjadi model terbaik (F1 0,9693). Sebagian besar kesalahan terjadi antara pesan penipuan dan promo.
Kata Kunci: Penipuan Digital, Klasifikasi Teks, XGBoost, Bayesian Optimization, SMOTE.
Downloads
References
Abdulhamid, S. M., Shuaib, M., Osho, O., Ismaila, I., & Alhassan, J. K. (2017). A Review on Mobile SMS Spam Filtering Techniques. IEEE Access, 5, 15650–15666. https://doi.org/10.1109/ACCESS.2017.2666785
Akiba, T., Sano, S., Yanase, T., Ohta, T., & Koyama, M. (2019). Optuna: A Next-generation Hyperparameter Optimization Framework. Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2623–2631. https://doi.org/10.1145/3292500.3330701
Almeida, T. A., Hidalgo, J. M. G., & Yamakami, A. (2011). Contributions to the Study of SMS Spam Filtering: New Collection and Results. Proceedings of the 11th ACM Symposium on Document Engineering, 259–262. https://doi.org/10.1145/2034691.2034742
Amin, M. B. M., Hakim, G., Maulana, M. T., Alwan, M. F., Anggraheni, H. S., Naufal, M. J., & Yudistira, N. (2024). Deteksi Spam Berbahasa Indonesia Berbasis Teks Menggunakan Model BERT. Jurnal Teknologi Informasi dan Ilmu Komputer, 11(6), 1291–1302. https://doi.org/10.25126/jtiik.2024118121
Bergstra, J., Bardenet, R., Bengio, Y., & Kégl, B. (2011). Algorithms for Hyper-Parameter Optimization. Advances in Neural Information Processing Systems, 24, 2546–2554.
Bergstra, J., & Bengio, Y. (2012). Random Search for Hyper-Parameter Optimization. Journal of Machine Learning Research, 13, 281–305.
Chawla, N. V., Bowyer, K. W., Hall, L. O., & Kegelmeyer, W. P. (2002). SMOTE: Synthetic Minority Over-sampling Technique. Journal of Artificial Intelligence Research, 16, 321–357. https://doi.org/10.1613/jair.953
Chen, T., & Guestrin, C. (2016). XGBoost: A Scalable Tree Boosting System. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 785–794. https://doi.org/10.1145/2939672.2939785
Herwanto, Chusna, N. L., & Arif, M. S. (2021). Klasifikasi SMS Spam Berbahasa Indonesia Menggunakan Algoritma Multinomial Naïve Bayes. Jurnal Media Informatika Budidarma, 5(4), 1316–1325. https://doi.org/10.30865/mib.v5i4.3119
Kompas.com. (2026, Juli 23). Komdigi Terima Laporan 30.000 Nomor Scamming Sepanjang Januari-Juli 2026. https://nasional.kompas.com/read/2026/07/23/17250811/komdigi-terima-laporan-30000-nomor-scamming-sepanjang-januari-juli-2026
Lemaître, G., Nogueira, F., & Aridas, C. K. (2017). Imbalanced-learn: A Python Toolbox to Tackle the Curse of Imbalanced Datasets in Machine Learning. Journal of Machine Learning Research, 18(17), 1–5.
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., et al. (2011). Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research, 12, 2825–2830.
Periskop.id. (2026, September 1). OJK Blokir Rp724,1 Miliar Dana Scam, 607 Ribu Rekening Dibekukan. https://periskop.id/keuangan/20260901/ojk-blokir-rp7241-miliar-dana-scam-607-ribu-rekening-dibekukan
Probst, P., Boulesteix, A.-L., & Bischl, B. (2019). Tunability: Importance of Hyperparameters of Machine Learning Algorithms. Journal of Machine Learning Research, 20(53), 1–32.
Saito, T., & Rehmsmeier, M. (2015). The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets. PLoS ONE, 10(3), e0118432. https://doi.org/10.1371/journal.pone.0118432
Snoek, J., Larochelle, H., & Adams, R. P. (2012). Practical Bayesian Optimization of Machine Learning Algorithms. Advances in Neural Information Processing Systems, 25, 2951–2959.
Wibisono, Y. (2018, Agustus 5). Dataset Klasifikasi Bahasa Indonesia (SMS Spam) & Klasifikasi Teks dengan Scikit-Learn. https://yudiwbs.wordpress.com/2018/08/05/dataset-klasifikasi-bahasa-indonesia-sms-spam-klasifikasi-teks-dengan-scikit-learn/
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Muhammad Ardiansyah Sembiring, Mustika Fitri Larasati Sibuea, Anjani Anjani (Author)
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
