Fractures are critical factors causing the degradation of mechanical properties in rock mass. In this study, addressing the damage effects and crack propagation problems induced by open fracture on rock mass, an analytical solution for the elastoplastic boundary equation at the tips of open fracture was derived under the consideration of the T-stress effect, based on the Mohr-Coulomb strength criterion. Using the plastic core region at the fracture tips as the evaluation indicator, we analyzed the influences of stress environment and fracture inclination angle (α) on rock mass damage. Through discrete element numerical simulations, the mesoscopic crack propagation characteristics and damage evolution patterns of open fracture were investigated. The obtained results indicate that: (1) When the lateral pressure coefficient (λ) is less than 0.4, the expansion rate of the plastic core region within the range [π/2, π] exceeds that of other regions as λ decreases, resulting in its morphology exhibiting an uneven distribution on the fracture surface; (2) For λ < 0.7, with α = 45° as the critical threshold, the plastic core region area decreases gradually when α decreases below the threshold, while it increases rapidly when α exceeds this value; (3) As α increases, the plastic core region area exhibits an S-shaped growth curve characterized by “slow growth–rapid growth–stabilization”, with steeper curve transitions observed at lower λ values; (4) A negative correlation exists between the uniaxial compressive strength of open-fractured rock mass and the plastic core region area. Specifically, an increase in plastic core region area reduces the required stress increment for crack propagation, extends crack length, and consequently decreases the compressive strength of the rock mass. The study reveals the mechanical mechanisms of rock mass damage caused by open fracture and provides a theoretical basis for rock mass stability assessment.