A holographic display system includes a light source that emits coherent light; a lateral displacement beam splitter that optically receives the coherent light and generates first reference light and second reference light; a first spatial light modulator (SLM) and a second SLM that optically receive the first reference light and the second reference light respectively, and construct first phase-only function (POF) light and second POF light respectively; a first beam splitter and a second beam splitter that optically receive the first POF light and the second POF light respectively, and generate first split light and second split light respectively; and a plurality of polarizers disposed between the first SLM and the first beam splitter, and between the second SLM and the second beam splitter, respectively.