aiqbench analysis
This research introduces the CPC-CMS framework, which utilizes expert judgment metrics like MCC and Kappa to systematically select optimal models for document-level sentiment analysis across social media datasets. The study compares classical algorithms against deep learning approaches such as ALBERT and LSTM, concluding that while transformers excel when latency is ignored, no single model dominates once time efficiency is factored in. For practitioners facing real-world deployment constraints, this work highlights the critical trade-off between raw predictive performance and computational speed without introducing new architectural innovations.
CPC-CMS: Cognitive Pairwise Comparison Classification Model Selection Framework for Document-level Sentiment Analysis
arXiv:2507.14022v2 Announce Type: replace Abstract: This study proposes the Cognitive Pairwise Comparison Classification Model Selection (CPC-CMS) framework for document-level sentiment analysis. The CPC, based on expert knowledge judgment, is used to calculate the weights of evaluation criteria, including accuracy, precision, recall, F1-score, Specificity, Matthews Correlation Coefficient (MCC), Cohen's Kappa (Kappa), and efficiency. Naive Bayes, Linear Support Vector Classification (LSVC), Random Forest, Logistic Regression, Extreme Gradient Boosting (XGBoost), Long Short-Term Memory (LSTM), and A Lite Bidirectional Encoder Representations from Transformers (ALBERT) are chosen as classification baseline models. A weighted decision matrix consisting of classification evaluation scores with respect to criteria weights is formed to select the best classification model for a classification problem. Three open datasets of social media are used to demonstrate the feasibility of the proposed CPC-CMS. Based on our simulation, for evaluation results excluding the time factor, ALBERT is the best for the three datasets; if time factor is included, no single model always performs better than the other models. With comparison, the conclusions are also supported by other aggregation and ranking methods including Analytic Hierarchy Process (AHP), Technique for Order of Preference by Similarity to Ideal Solution (TOPSIS) and Multi-Objective Optimization by Ratio Analysis (MOORA), although aggregation values and ranks may be different. The CPC-CMS can be applied to the other classification applications in different areas.
Abstract:This study proposes the Cognitive Pairwise Comparison Classification Model Selection (CPC-CMS) framework for document-level sentiment analysis. The CPC, based on expert knowledge judgment, is used to calculate the weights of evaluation criteria, including accuracy, precision, recall, F1-score, Specificity, Matthews Correlation Coefficient (MCC), Cohen's Kappa (Kappa), and efficiency. Naive Bayes, Linear Support Vector Classification (LSVC), Random Forest, Logistic Regression, Extreme Gradient Boosting (XGBoost), Long Short-Term Memory (LSTM), and A Lite Bidirectional Encoder Representations from Transformers (ALBERT) are chosen as classification baseline models. A weighted decision matrix consisting of classification evaluation scores with respect to criteria weights is formed to select the best classification model for a classification problem. Three open datasets of social media are used to demonstrate the feasibility of the proposed CPC-CMS. Based on our simulation, for evaluation results excluding the time factor, ALBERT is the best for the three datasets; if time factor is included, no single model always performs better than the other models. With comparison, the conclusions are also supported by other aggregation and ranking methods including Analytic Hierarchy Process (AHP), Technique for Order of Preference by Similarity to Ideal Solution (TOPSIS) and Multi-Objective Optimization by Ratio Analysis (MOORA), although aggregation values and ranks may be different. The CPC-CMS can be applied to the other classification applications in different areas.
Computation and Language (cs.CL); Machine Learning (cs.LG)
arXiv:2507.14022 [cs.CL]
(or arXiv:2507.14022v2 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2507.14022 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 310 words
