A voice signal processing apparatus and a voice signal processing method are provided. A first sampling point of an mth original frequency-lowered signal frame phase-matched to the sampling point corresponding to a phase reference sampling point number is determined according to the phase reference sampling point number of an (m−1)th original frequency-lowered signal frame corresponding to a middle sampling point of an (m−1)th renovating frequency-lowered signal frame. The q consecutive sampling points starting from the first sampling point are used as the sampling points of an mth renovating frequency-lowered signal frame.