A prism unit includes: a color synthesizing prism; a fixing member having a linear expansion coefficient different from a linear expansion coefficient of the color synthesizing prism; and a first adhesive layer and a second adhesive layer, wherein the color synthesizing prism is adhered to the fixing member via the first adhesive layer and the second adhesive layer which are arranged in this order sequentially from the fixing member, and the second adhesive layer has a lower tensile strength than the first adhesive layer.