A method of detecting a trailer angle of a trailer being towed by a vehicle includes capturing frames of image data via a camera of the vehicle while the vehicle is towing a trailer, and, via processing by an image processor of frames of captured image data while the vehicle is towing the trailer, determining presence of the trailer being towed by the vehicle, and determining feature vectors of the trailer being towed by the vehicle. The feature vectors of the trailer are determined by determining features that have similar angular changes between a current frame of captured image data and a previous frame of captured image data. The angle of the trailer relative to the vehicle is determined responsive to tracking of determined feature vectors of the trailer present in the field of view of the camera over multiple frames of captured image data.