A contact model considering rolling resistance is introduced and implemented into the distinct element method (DEM) for analyzing the earth pressure at critical state against a rigid wall. The variation of earth pressure coefficient and the shear band formation in the backfill material are analyzed, when the passive earth pressure varies with displacement for the retaining wall with different densities and different displacement modes (passive T mode, RB mode, RT mode). The DEM simulation results are compared with those of other researches. It is shown that earth pressure coefficient is greatly affected by different void ratios and wall displacement modes. The -displacement relation switches from hardening to softening when the void ratio decreases or the relative density increases. In contrast, the -displacement relation demonstrates either hardening or softening feature for the medium dense backfills even they exhibit a strain softening response in the biaxial compression tests. As the retaining wall moves to soil behind the wall, the shear strain increases and the strain localization occurs, i.e. shear band forms in the soil behind the wall. Similar to the shear strain field from laboratory experiment, the distributions of the shear band in the critical state are better simulated by DEM. Meanwhile, the surface of the backfill is no long a smooth surface rather than an upheaval surface, the value of upheaval increases significantly with the increase of the displacements of retaining wall, and finally the soil body behind the wall damages.