A vehicular trailer assist system includes a camera at a vehicle and viewing at least a portion of a trailer hitched to the vehicle. A control, responsive to processing of frames of image data captured by the camera and during a calibration maneuver by the vehicle, determines an initial trailer template of trailer hitched to the vehicle. The control, during a turning portion of the calibration maneuver, and at least in part via processing of frames of image data captured by the camera, determines a hitch ball location of a hitch of the vehicle based on the determined initial trailer template. The control, after completion of the calibration maneuver, determines a current trailer angle of the trailer relative to the vehicle via processing of frames of image data captured by the camera and based on the determined hitch ball location as the vehicle is driven along a road.