The distribution and value of lateral earth pressure are closely related to the displacement mode of retaining wall. The critical depth of plastic zone behind the retaining wall should be considered during design when the backfill is in active earth pressure state. Based on the upper bound method of limit analysis, the infinitesimal backfill body under elastic overburden and adjacent wall are studied to determine the external work rate and internal energy dissipation rate. The equations are obtained to calculate the critical depth of plastic zone under sliding mode, considering the effect of displacement direction of the retaining wall. The results show that Rankine and other theories are special cases of the recommended method. The critical depth of plastic zone increases with the increase of the angle between the displacement direction of wall and the horizontal plane, and with the increase of the values of cohesion and friction angle of wall-soil interface and the values of cohesion and friction angle of backfill. However, when the wall displaces horizontally, the critical depth of plastic zone becomes constant regardless of the change of friction angle of wall-soil interface.