AutoCluster, AutoTopicModeling, AutoTrendAnalysis: A Complete AutoML Pipeline for Predicting Emerging Trends

Abstract:Predicting emerging trends is vital for businesses, researchers, and policymakers; yet traditional approaches often lack scalability and adaptability. This paper presents a trend prediction framework based on Automated Machine Learning (AutoML), designed to extract insights from textual datasets with temporal attributes. The system ingests subject-specific textual entries accompanied by a date field. The pipeline begins with preprocessing and embedding, followed by AutoClustering, which uses meta-learning to select the optimal clustering algorithm. AutoTopicModeling then applies successive halving to identify the best topic modeling method: Latent Dirichlet Allocation (LDA), Latent Semantic Analysis (LSA), BERTopic, or Non-negative Matrix Factorization (NMF) based on the coherence score for each cluster. For trend forecasting, AutoTrendAnalysis evaluates multiple models: Facebook Prophet, AutoRegressive Integrated Moving Average (ARIMA), Seasonal-Trend decomposition using Loess (STL), and Long Short-Term Memory (LSTM) selecting the most accurate based on Root Mean Square Error (RMSE), either through successive halving or exhaustive comparison. Topics are classified as strong signals, weak signals, or noise based on forecasting outcomes, enabling the identification of emerging trends. By automating clustering, topic modeling, and time series forecasting, this research enhances trend prediction accuracy while reducing manual effort. The proposed system offers a scalable and user-friendly solution suitable for real-time applications and stakeholders with limited machine learning expertise. Experimental results demonstrate that the proposed system's best trial achieves a final RMSE of 7.099, indicating high predictive accuracy.

Digital Libraries (cs.DL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)

arXiv:2607.22641 [cs.DL]

(or arXiv:2607.22641v1 [cs.DL] for this version)

https://doi.org/10.48550/arXiv.2607.22641 Focus to learn more arXiv-issued DOI via DataCite (pending registration)

https://doi.org/10.1109/ICCA66035.2025.11430858 Focus to learn more DOI(s) linking to related resources

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.