Provide are an optical element that can improve a utilization efficiency of light while increasing an optical path length, an image display unit, and a head-mounted display. The optical element includes, in the following order: a first absorptive linearly polarizing plate; a first reflective linearly polarizing plate; a first retardation plate; a partially reflecting mirror; a second retardation plate; and a second reflective linearly polarizing plate, in which a turning direction of circularly polarized light that is reflected from the first reflective linearly polarizing plate in a case where light transmits through the first retardation plate and is incident into the first reflective linearly polarizing plate is opposite to a turning direction of circularly polarized light that is reflected from the second reflective linearly polarizing plate in a case where light transmits through the second retardation plate and is incident into the second reflective linearly polarizing plate.