Clay mixed with gravel is becoming widely used as a core material of high earth and rockfill dams. Its tensile fracture characteristics are crucial to the safety of the core wall. Clay mixed with gravel is considered as a four-phase composite material and a mesoscopic numerical model is constructed via image processing and random packing to consider the real shape and distribution of gravels. With this model, whose effectiveness is validated by physical tests, the macroscopic and mesoscopic tensile fracture characteristics of clay mixed with gravel are analyzed. The nonlinear behavior on macro scale is due to mesoscopic heterogeneity. The addition of gravel greatly decreases the tensile strength but increases the post-peak tensile resistance. Soil-rock interfaces are the source of microcracks and tensile fracture is a process where microcracks initiate, propagate and coalesce to macrocrack. Tensile fracture occurs in a region, i.e. the fracture process zone, and the formation of this zone is related to the random distribution of gravels. Through numerical experiments, the influence of the random packing and content of gravel on the macroscopic and mesoscopic behaviors of clay mixed with gravel is analyzed.