According to an embodiment, a signal processing apparatus includes an ambient sound estimating unit, a representative component estimating unit, a voice estimating unit, and a filter generating unit. The ambient sound estimating unit is configured to estimate, from the feature, an ambient sound component that is non-stationary among ambient sound components having a feature. The representative component estimating unit is configured to estimate a representative component representing ambient sound components estimated from one or more features for a time period, based on a largest value among the ambient sound components within the time period. The voice estimating unit is configured to estimate, from the feature, a voice component having the feature. The filter generating unit is configured to generate a filter for extracting a voice component and an ambient sound component from the feature, based on the voice component and the representative component.