A light source device has a first monochromatic light source group, a second monochromatic light source group, a polarization selection and wavelength selection element, a first phosphor layer, and a second phosphor layer. The first monochromatic light source group outputs first polarized light having a fixed polarization direction. The second monochromatic light source group outputs second polarized light having a fixed polarization direction. The first phosphor layer emits light in a first wavelength band. The second phosphor layer emits light in a second wavelength band. The polarization and wavelength selection element directs the first polarized light onto the first phosphor layer by transmitting the first polarized light and directs the second polarized light onto the second phosphor layer by reflecting the second polarized light, reflects the light in the first wavelength band and transmits the light in the second wavelength band.