A three-dimensional (3D) discrete element model (DEM) is established to simulate micro- and macro-scale mechanical behaviors of railway ballast. In this model, the clump models with irregular shape are built to simulate the real ballast particle geometry morphology captured by using a 3D laser scanner. A full-scale model of high-speed railway ballasted track is conducted to validate the DEM model. The deformation and vibration response of railway ballast are obtained under the static load and the cyclic load in the laboratory. The amplitudes of the ballast acceleration and its attenuation law with the depth acquired by DEM simulation and experimental tests are close. Numerical results of the vertical deformation of railway ballast under the static loads and the permanent deformation under cyclic loads are good agreement with experimental data, which demonstrates that the established DEM model is reasonable. Therefore, the obtained vibration response and deformation of railway are reliable and relatively accurate. This DEM model can be used for subsequent analysis on the micro- and macro- mechanical behaviors and deformation of high-speed railway ballast.