A pogo pin connector includes an insulating housing, a plurality of pogo pins and contact terminals assembled forward in the insulating housing respectively. Front ends of the pogo pins retractably project beyond a front face of the insulating housing. A rear end of each of the pogo pins protrudes outward to form a riveted flange. Each of the contact terminals has a fastening plate placed levelly, an elastic contact arm curvedly extending upward from a front end of the fastening plate to elastically project beyond a top face of the insulating housing, and a fixing strip extending downward from a rear end of the fastening plate with a through hole being opened in a bottom end thereof. The rear ends of the pogo pins pass through the through holes of the contact terminals respectively to make the riveted flanges and the corresponding fixing strips riveted together.