Abstract: There are various complex rock mass instability risk disasters commonly faced in underground mining processes, such as frequent ground pressure disasters, rock mass instability caused by blasting construction disturbances, and roof subsidence deformation. To precisely enhance the level of mine safety production management, this paper took the Jinchuan Second Mining Area as the specific research object and systematically sorted out the relevant engineering geological data of the rock mass in this mining area. From the database, 6 912 representative data samples were selected and obtained as the research foundation. The study employed four mainstream machine learning algorithms: GBDT, XGBoost, LightGBM, and CatBoost, to construct a rock mass instability risk classification and evaluation model. This model conducted a comprehensive and systematic analysis and evaluation of the rock mass instability risk situation in the Jinchuan Second Mining Area. To objectively and scientifically verify the performance of the model, four core evaluation indicators, namely precision, recall, F1 score, and accuracy, were selected to quantitatively assess the prediction performance of the aforementioned four models. The research results indicate that the constructed four machine learning models exhibit good applicability and reliability in rock mass instability risk evaluation, effectively achieving precise identification and classification of rock mass instability risks. Among them, the model based on the LightGBM algorithm performs optimally in all evaluation indicators, demonstrating the best comprehensive evaluation performance. This research outcome provides a scientific and reliable reference for precise analysis, risk warning, and safety production management decision-making of rock mass instability risk levels in the Jinchuan Second Mining Area and similar underground mines. It holds significant practical importance for ensuring the safety of mining operations.