Fracture toughness of rocks characterizes the resistance to crack initiation and propagation. As an important index of mechanical property, it plays an irreplaceable role in the theoretical research of rock mechanics and practical application to rock engineering problems. Due to the complexity of rock structures, most of rock fractures occur under mixed mode loading conditions. Therefore, it is important to investigate the mixed mode fracture toughness of rocks. Rust rocks are widely used in the construction, and thus are selected to study the fracture behavior. A series of radial compression tests is carried out on 18 cracked straight through Brazilian disc (CSTBD) specimens to obtain the pure mode I, pure mode II and mixed mode I/II fracture toughness. Then experimental results are compared with theoretical values calculated by the generalized maximum tangential stress (GMTS) criterion. The results show that the values of fracture toughness for pure mode I and pure mode II are 1.01 MPa?m0.5 and 1.51 MPa?m0.5, respectively. The fracture toughness of pure mode II is 1.49 times higher than that of the pure mode I. The fracture toughness of pure mode II is close to the value of 1.34 obtained by the GMTS criterion, but is larger than the value of 0.87 achieved by the maximum tangential stress (MTS) criterion. The T-stress and the critical distance (rc) related to the size of fracture process zone in front of the crack tip have a great influence on crack propagation path and mixed mode fracture toughness of rock materials. As the generalized maximum tangential stress (GMTS) criterion takes into account the T-stress, it has a good agreement with experimental results.