Abstract: A principle of nonlinear relationship proposed by Hoek is elaborated, then based on that, a new algorithm of obtaining shear strength of rock mass is brought forward by considering the overall or average level when rock mass is broken. Through the new algorithm, the magnitudes of shear stress can be calculated at different given normal stresses, and a series of shear stress and normal stress are generated; then the values of cohesion and internal frictional angle are determined via linear regression analysis, which solves the determination of shear strength under certain normal stress that Hoek-Brown criterion fitting algorithm can not do. Then a research on shear strength of rock mass from China-Myanmar oil and gas pipelines (domestic section) Lancang River across domain engineering as an example and a further comparison study is taken. It is shown that the values of cohesion and internal frictional angle determined by the first method according to the new algorithm are little different to these gained through the Hoek-Brown criterion fitting algorithm. The reason is that the factors considered are not the same, while the normal stress is in considered by the two algorithms resulting in confining pressure and normal stress are varied practical engineering is low, the cohesion and internal friction angle proposed by the second method according to the new algorithm, the results obtained are close to the actural situation, which has overcome the problem that Hoek-Brown criterion fitting algorithm relies on the confining pressure.

Key words: rock mechanics, shear strength, nonlinear relationship, shear stress, normal stress, engineering application