Invention Grant
- Patent Title: Audio-visual speech enhancement
-
Application No.: US16783021Application Date: 2020-02-05
-
Publication No.: US11244696B2Publication Date: 2022-02-08
- Inventor: Kazuhito Koishida , Michael Iuzzolino
- Applicant: Microsoft Technology Licensing, LLC
- Applicant Address: US WA Redmond
- Assignee: Microsoft Technology Licensing, LLC
- Current Assignee: Microsoft Technology Licensing, LLC
- Current Assignee Address: US WA Redmond
- Agency: Alleman Hall Creasman & Tuttle LLP
- Main IPC: G10L21/0364
- IPC: G10L21/0364 ; G06K9/00 ; G10L15/22 ; G10L15/25 ; G10L25/18

Abstract:
Example speech enhancement systems include a spatio-temporal residual network configured to receive video data containing a target speaker and extract visual features from the video data, an autoencoder configured to receive input of an audio spectrogram and extract audio features from the audio spectrogram, and a squeeze-excitation fusion block configured to receive input of visual features from a layer of the spatio-temporal residual network and input of audio features from a layer of the autoencoder, and to provide an output to the decoder of the autoencoder. The decoder is configured to output a mask configured based upon the fusion of audio features and visual features by the squeeze-excitation fusion block, and the instructions are executable to apply the mask to the audio spectrogram to generate an enhanced magnitude spectrogram, and to reconstruct an enhanced waveform from the enhanced magnitude spectrogram.
Public/Granted literature
- US20210134312A1 AUDIO-VISUAL SPEECH ENHANCEMENT Public/Granted day:2021-05-06
Information query