An interior rearview mirror assembly for a vehicle includes a mirror head, a mirror mounting structure configured to adjustably mount the mirror head at an interior portion of the vehicle, and a receiving portion for receiving a portable accessory module. A retaining element is configured to attach at the mirror casing at the receiving portion to releasably secure the portable accessory module at the receiving portion. With the retaining element at least partially detached from the mirror casing, the portable accessory module is insertable into and removable from the receiving portion. With the portable accessory module inserted into and received at the receiving portion and with the retaining element attached at the mirror casing, the retaining element overlaps a lower portion of the portable accessory module to secure the portable accessory module at the receiving portion.