A voice signal processing apparatus and a voice signal processing method are provided. A last sampling point of an mth original frequency-lowered signal frame is determined according to a phase reference sampling point number of the mth original frequency-lowered signal frame. Here, the phase reference sampling point number corresponds to a middle sampling point of an mth renovating frequency-lowered signal frame, and the last sampling point is phase-matched with a sampling point corresponding to the phase reference sampling point number in the mth original frequency-lowered signal frame. P consecutive sampling points starting from the last sampling point are applied as sampling points of an (m+1)th renovating frequency-lowered signal frame.