Comparing KNN, Logistic Regression, Random Forest and BERT Fine-Tuning for Scam Message Detection

Abstract

Scam message detection remains a persistent challenge, particularly with the rise of adversarial content crafted to bypass traditional filters. This study compares the effectiveness of conventional classifiers—K-Nearest Neighbors (KNN), Logistic Regression, and Random Forest—using frozen BERT embeddings, against a fully fine-tuned BERT model trained end-to-end. The evaluation is conducted on a labeled dataset containing regular scam messages, adversarial scam messages generated by large language models, and legitimate non-scam texts. Among the tested models, the fine-tuned BERT achieves the highest multiclass classification accuracy of 95.83% and binary classification accuracy of 96.67%. Logistic regression also reaches 96.67% binary accuracy, offering a lightweight and computationally efficient alternative. Visualizations of attention matrices reveal that fine-tuning improves model interpretability by concentrating attention on task-relevant tokens in deeper transformer layers. These findings suggest a practical trade-off between model complexity, interpretability, and performance. While fine-tuning offers superior accuracy and insight into model behavior, traditional methods remain valuable in resource-constrained or time-sensitive scenarios. This work provides empirical evidence and visual analysis to guide the selection of text classification strategies in adversarial environments, contributing to more robust and explainable approaches for real-world scam detection.