A virtual image display device includes a light modulating element including a liquid crystal pixel and a light transmitting region, a light shielding member being arranged on an external side of the light modulating element, a backlight being arranged between the light modulating element and the light shielding member and being configured to irradiate the liquid crystal pixel, a polarizing plate configured to limit external light that passes through the light transmitting region of the light modulating element and image light that is emitted from the liquid crystal pixel, to polarized light, and cause the light to pass therethrough, an image selection conversion member configured to selectively change a polarization direction of the image light emitted from the polarizing plate, and a polarization separation lens element being arranged on the face side of the image selection conversion member.