A diffractive optical element includes a first optical member having a first diffraction grating with a sawtooth cross section; and a second optical member which has a second diffraction grating having a sawtooth cross section and which has a refractive index different from that of the first optical member. In the diffractive optical element, the first and second optical members are stacked so that the first and second diffraction gratings closely contact each other. Arithmetical mean roughness of a diffraction surface of the first diffraction grating is set so as to fall within a predetermined range.