A rivet setting device contains in a housing a receiving chamber in which a break-off rivet mandrel is situated after the setting operation. Arranged in this receiving chamber are two elongate magnets which leave free between them a space for the residual mandrel and which attract the residual mandrel from the holder for the rivet after it has been broken off and retain it. The holder for the residual mandrel can be moved with a transverse component in the device, with the residual mandrel being moved past a stripping element into a region outside the axis of the holder for the rivet. During the rearward movement of the residual mandrel holder, the residual mandrel is stripped off and remains in a collecting chamber.