An audio jack connector has an insulating housing. The insulating housing has an inserting hole extending from a front to a rear for receiving the audio plug connector and a plurality of recesses communicating with the inserting hole for receiving a plurality of terminals. An elastic body which is disposed in the inserting hole perpendicular to an extending direction of the inserting hole has two opposite tightening arms and a connecting portion interconnected with opposite ends of the tightening arms. A front end of the audio plug connector passes through the elastic body and is restrained by the elastic body. Two tightening arms of the elastic body are against the portions of the front end of the audio plug connector from opposite directions when the audio plug connecter is inserted into the inserting hole of the insulating housing.