Intent Mining of Thai Phone Call Text Using a Stacking Ensemble Classifier with GPT-3 Embeddings
Article Sidebar
Main Article Content
Abstract
Intent mining has recently attracted Natural Language Processing (NLP) research communities. Despite the extensive research on English and other widely spoken languages, intent mining in Thai remains unexplored. This paper proposes an extended framework for mining intentions in Thai phone call text. It utilized a stacking ensemble method with GPT-3 embeddings, constructed by systematically determining based and meta-classifiers using Q-statistic and F1 scores. Overall, the based classifiers consisting of Support Vector Classier (SVC), k-nearest Neighbors (KNN), and Random Forest (RF) were derived with a meta-classier, Logistic Regression (LR). We compared the mining results, derived through the proposed Stacking Ensemble Classier (SEC), to 1) the individual base classifiers and 2) the three BERT baselines: BERT Multilingual Uncased, and BERT-th, and BERT Based EN-TH Cased. The results revealed that SEC could outperform SVC, KNN, RF, BERT Multilingual Uncased, and BERT-th, except BERT Based EN-TH Cased. However, a statistical analysis conducted using Friedman and Holm's post hoc tests reported no statistically significant difference between SEC and BERT Based EN-TH Cased, inferring that the two classifiers perform similarly practically.
Article Details
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
References
R. C. Schank and R. P. Abelson, “Scripts, Plans, Goals, and Understanding,” Hillsdale, New Jersey: Lawrence Erlbaum Associates, 1997.
G. Khodabandelou, C. Hug, R. Deneck`ere and C. Salinesi, “Process Mining Versus Intention Mining,” Proceedings of BPMDS EMMSAD 2013, Enterprise, Business-Process and Information Systems Modeling, pp. 466-480, 2013.
J. W. Duan, Y. H. Chen, T. Liu and X. Ding, “Mining Intention-Related Products on Online Q&A Community,” Journal of Computer Science and Technology, vol. 30, pp. 1054-1062, 2015.
G. Khodabandelou, C. Hug, R. Deneckere and C. Salinesi, “Supervised Intentional Process Models Discovery Using Hidden Markov Models,” Proceedings of the IEEE 7th International Conference on Research Challenges in Information Science (RCIS), pp. 1–11, May 2013.
V. D. Jadhav and S. N. Deshmukh, “Mining Movie Intention Using Bayes and Maximum Entropy Classifiers,” International Journal of Computer Applications, vol. 975, no. 8887, pp. 8-19, 2016.
M. Hamroun and M. S. Gouider, “A Survey on Intention Analysis: Successful Approaches and Open Challenges,” Journal of Intelligent Information Systems, vol. 55, pp. 423-443, 2020.
A. Rashid, M. S. Farooq, A. Abid, T. Umer, A. K. Bashir and Y. B. Zikria, “Social Media Intention Mining for Sustainable Information Systems: Categories, Taxonomy, Datasets and Challenges,” Complex Intelligent Systems, vol. 9, pp. 2773–2799, 2021.
N. Sanchan, “Mining Users’ Intentions from Thai Tweets Using BERT Models,” Journal of Information Science and Technology, vol. 13, no. 1, pp. 17-25, 2023.
A. De and S. K. Kopparapu,“A Rule-Based Short Query Intent Identification System,” Proceedings of the 2010 International Conference on Signal and Image Processing, pp. 212-216, 2010.
O. A. Chidinma, S. Borah and R. Panigrahi, “Suicidal Intent Prediction Using Natural Language Processing (Bag of Words) Approach,” Proceedings of the International Conference on Computing and Communication (IC3 2020), pp. 147-153, 2021.
H. D. Rebelo, L. A. de Oliveira, G. M. Almeida, C. A. Sotomayor, G. L. Rochocz and W. E. Melo, “Intent Identification in Unattended Customer Queries Using an Unsupervised Approach,” Journal of Information & Knowledge Management, vol. 20, no. 3, pp. 1-26, 2021.
G. Forman, H. Nachlieli and R. Keshet, “Clustering by Intent: A Semi-Supervised Method to Discover Relevant Clusters Incrementally,” Proceedings of the European Conference on Machine Learning and Knowledge Discovery in Databases (ECML PKDD 2015), pp. 20-36, 2015.
J. Wang, G. Cong, X. Zhao and X. Li, “Mining User Intents in Twitter: A Semi-Supervised Approach to Inferring Intent Categories for Tweets,” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 29, no. 1, 2015.
J. Xu, J. Wang, Y. Tian, J. Yan and X. Li, “SE-stacking: Improving User Purchase Behavior Prediction by Information Fusion and Ensemble Learning,” PLOS ONE, vol. 15, no. 11, pp. e0242629, 2020.
Y. Wang, S. Liu, S. Li, J. Duan, Z. Hou, J. Yu and K. Ma, “Stacking-Based Ensemble Learning of Self-Media Data for Marketing Intention Detection,” Future Internet, vol. 11, no. 7, p. 155, 2019.
M. Li, L. Yang, W. Yixuan and Q. Zhang, “Predicting User Pay Conversion Intention Based on Stacking Ensemble Learning: Case Study of Free Value-Added Games,” Data Analysis and Knowledge Discovery, vol. 8, no. 2, pp. 143–154, 2023.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser and I. Polosukhin, “Attention Is All You Need,” Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 6000-6010, 2017.
J. Devlin, M. W. Chang, K. Lee and K. Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4171-4186, 2019.
C. Oswald, S. E. Simon and A. Bhattacharya, “Spotspam: Intention Analysis-Driven SMS Spam Detection Using BERT Embeddings,” ACM Transactions on the Web (TWEB), vol. 16, no. 3, pp. 1-27, 2022.
F. Fern´andez-Mart´ınez, C. Luna-Jim´enez, R. Kleinlein, D. Griol, Z. Callejas and J. M. Montero, “Fine-Tuning BERT Models for Intent Recognition Using a Frequency Cut-Off Strategy for Domain-Specific Vocabulary Extension,” Applied Sciences, vol. 12, no. 3, p. 1610, 2022.
T. W. Wu, R. Su and B. Juang, “A Label-Aware BERT Attention Network for Zero-Shot Multi-Intent Detection in Spoken Language Understanding,” Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 4884-4896, 2021.
T. Brown et al., “Language Models Are Few-Shot Learners,” Proceedings of the 34th International Conference on Neural Information Processing Systems, pp. 1811-1911, 2020.
X. Song, K. He, P. Wang, G. Dong, Y. Mou, J. Wang, Y. Xian, X. Cai, and W. Xu, “Large Language Models Meet Open-World Intent Discovery and Recognition: An Evaluation of Chat-GPT,” Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 10291-10304, 2023.
Z. Li, S. Larson, and K. Leach, “Generating Hard-Negative Out-of-Scope Data with Chat-GPT for Intent Classification,” Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pp.7634-7646, 2024.
A. Daowraeng, “Text Classification in Thai Stacking Model,” Accessed Nov. 21, 2024, [Online.] Available from: https://medium.com/super-ai-engineer/การทำ-text-classification-ภาษาไทยด้วย-stacking-model-be12defb989e.
K. Viriyayudhakorn, “Truevoice-intent.GitHub,” Accessed February 15, 2024, [Online.] Available from:https://github.com/kobkrit/truevoice-intent.
P. Kaewnoo and T. Senivongse, “Identification of Software Problem Report Types Using Multiclass Classification,” Proceedings of the 2019 3rd International Conference on Software and eBusiness, pp. 104-109, 2019.
K. Shah, H. Patel, D. Sanghvi and M. Shah, “A Comparative Analysis of Logistic Regression, Random Forest and KNN Models for the Text Classification,” Augmented Human Research, vol. 5, no. 1, p. 12, 2020.
L. I. Kuncheva and C. J. Whitaker, “Measures of Diversity in Classifier Ensembles and Their Relationship with the Ensemble Accuracy,” Machine Learning, vol. 51, pp. 181–207, 2003.
J. Demˇsar, “Statistical Comparisons of Classifiers over Multiple Data Sets,” The Journal of Machine Learning Research, vol. 7, pp. 1-30,2006.
M. Abdel-Basset, H. Hawash, K. A. Alnowibet, A. W. Mohamed and K. M. Sallam, “Interpretable Deep Learning for Discriminating Pneumonia from Lung Ultrasounds,” Mathematics, vol. 10, no. 21, p. 4153, 2022.
S. Holm, “A Simple Sequentially Rejective Multiple Test Procedure,” Scandinavian Journal of Statistics, vol. 6, no. 2, pp. 65-70, 1979.
N. Sanchan, “Comparative Study on Automated Reference Summary Generation Using BERT Models and ROUGE Score Assessment,” Journal of Current Science and Technology, vol. 14, no. 2, 2024.
A. Abdaoui, C. Pradel and G. Sigel, “Load What You Need: Smaller Versions of Multilingual BERT,” Proceedings of SustaiNLP: Workshop on Simple and Efficient Natural Language Processing, pp. 119-123, 2020.