A vision system of a vehicle includes an interior monitoring system operable to determine a gaze direction of the driver of the vehicle. A plurality of cameras is disposed at a vehicle and the cameras have respective fields of view exterior of the vehicle. A heads up display system is operable to display virtual images for viewing by the driver of the vehicle, with the displayed virtual images being displayed in the driver's gaze direction, such as via heads up display glasses worn by the driver. Responsive to a determination of the driver gazing in a direction towards a non-transparent portion of the vehicle, the heads up display system displays an image derived from image data captured by at least one of the plurality of cameras to display a virtual image representative of the driver's view as it would be through the non-transparent portion of the vehicle.