A polarizing beam splitter assembly for directing image light on an input path into multiple exit light paths comprises multiple prisms with edges that meet to form a seam. The polarizing beam splitter assembly includes a diffracting element prior to the seam in the input light path. The diffracting element comprises a geometry that performs at least one of blocking a portion of the image light and scattering a portion of the image light.