An ultrasonic gesture recognition system is provided that recognizes gestures based on analysis of return signals of an ultrasonic pulse that is reflected from a gesture. The system transmits an ultrasonic chirp and samples a microphone array at sample intervals to collect a return signal for each microphone. The system then applies a beamforming technique to frequency domain representations of the return signals to generate an acoustic image with a beamformed return signal for multiple directions. The system then generates a feature image from the acoustic images to identify, for example, distance or depth from the microphone array to the gesture for each direction. The system then submits the feature image to a deep learning system to classify the gesture.