An aberration measurement system includes an interferometer that includes a polarization adjuster configured to adjust a polarization plane of coherent light, and a phase shifter configured to shift a phase of reference light. The aberration measurement system further includes a signal processing unit configured to obtain a nonpolarization aberration of the test object, and coefficients a=sin ε cos 2θ, b=sin ε sin 2θ, and c=cos ε for a retardation amount 2ε and a principal axis direction θ of a polarization aberration of the test object, based on data of a plurality of interference pattern images which provide at least three complex visibilities obtained from the interferometer after a polarization adjuster adjusts the polarization plane of the coherent light, and to determine signs of the coefficients a, b, and c based on the nonpolarization aberration.