There is provided a method of processing an ultrasound spectrum image. According to such method, a spectrum image is formed based on ultrasound data and then the noise is removed from the spectrum image. The noise-removed spectrum image is matched with one or more spectrum models representing specific spectrum types. Then, whether or not the noise-removed spectrum image contains an aliasing is checked. If the noise-removed spectrum image contains the aliasing, then the aliasing is removed from the noise-removed spectrum image to provide a noise-removed spectrum image without the aliasing. Thereafter, contour tracing is performed on the noise-removed spectrum image without the aliasing to detect contour points. Further, peak tracing is performed on the noise-removed spectrum image without the aliasing to detect peaks.