In order to predict the deformation characteristics of retaining structure in deep foundation pit excavation, we study the design method of subway station excavation retaining structure in soft soil foundation pit from a retaining structure’ system stiffness point of view. A new concept of system stiffness, i.e. multiple variables system stiffness (MVSS) is proposed in view of the defects of Clough composite stiffness model. In addition to the stiffness of the retaining wall and the vertical spacing between the supports, as included in the original Clough model, several important parameters are introduced in the proposed MVSS model, including the depth of the foundation pit, the bracing stiffness, the horizontal spacing between the supports, basement reinforcement. Finite element method is employed to calculate the maximal lateral displacement of the retaining structure. The results show that the proposed MVSS model can describe excellently the deformation of the retaining structure. The measured displacement data from several foundation pits also validates the applicability of proposed method. Therefore, the proposed model is a liable and valuable way to determine the system stiffness of the deep foundation pit of subway in different strata. There exists a negative correlation between retaining structure of foundation pit and maximum lateral deformation, excavation MVSS comprehensive stiffness, but when the MVSS comprehensive stiffness increases up to a certain degree, the continued increase of MVSS comprehensive stiffness has an insignificant effect on controlling further the deformation of the retaining structure of foundation pit.