A backlight adjusting system and method is adapted for adjusting a backlight brightness of a light emitting element for a display panel based on an ambient light around the display panel. The backlight adjusting system includes a first calculator, a processor and a driver. The first calculator calculates a target display brightness value of the display panel based on an actual ambient brightness value of the ambient light, a predetermined optimal ambient brightness value of the ambient light, and a predetermined optimal display brightness value of the display panel determined according to the predetermined optimal ambient brightness value. The processor obtains a target pulse duty value based on the target display brightness value. The driver generates a driving signal based on the target pulse duty value utilizing pulse width modulation techniques for driving the light emitting element so as to adjust the backlight brightness of the light emitting element.