To evaluate the effect of the intermediate principal stress on the ultimate strength of rock, a new three-dimensional (3D) Hoek-Brown failure criterion is established by adding a quantitative representation of the intermediate principal stress to the power-law item of the generalized Hoek-Brown failure criterion. The new failure criterion in principal stress space is a curved hexahedron circumscribing the envelope of the Hoek-Brown failure criterion through three angular points. It is a nonlinear power-law curve in the plane. Comparative study with three different rock failure criteria i.e. Hoek-Brown criterion, Drucker-Prager criterion, and Mogi (1971) criterion is conducted by fitting the triaxial test data of four different rocks. Then the applicability of four failure criteria is discussed on describing the effect of intermediate principal stress. The results show that the new 3D Hoek-Brown failure criterion fits the triaxial strength data best, followed by the Mogi (1971) criterion, whereas the Hoek-Brown criterion and the Drucker-Prager criterion have a relatively poor performance. Therefore, the new 3D Hoek-Brown failure criterion is the most suitable to predict the true triaxial strength and to describe the effect of the intermediate principal stress on trachy, marble, granite and other hard brittle rocks.