A vehicular driver monitoring system includes an interior rearview mirror assembly having a mirror head accommodating a mirror reflective element and adjustable about a mounting structure. The mirror head is adjustable relative to the mounting structure to adjust a rearward view provided by the mirror reflective element. A driver monitoring camera is accommodated by the mirror head behind the mirror reflective element. With the mirror head adjusted to provide the rearward view, a normal vector extending perpendicular to the mirror reflective element is directed toward a driver side of the cabin of the vehicle and, responsive to determination that a passenger is present at a passenger side of the cabin of the vehicle, a principal viewing axis of the driver monitoring camera is angled more toward a centerline axis of the vehicle than the normal vector of the mirror reflective element.