A method of audio source separation includes steps of applying a demixing matrix on a plurality of received signals to generate a plurality of separated results; performing a recognition operation on the plurality of separated results to generate a plurality of recognition scores; generating a constraint according to the plurality of recognition scores; and adjusting the demixing matrix according to the constraint; where the adjusted demixing matrix is applied to the plurality of received signals to generate a plurality of updated separated results from the plurality of received signals.