A headphone controlling system is used in a portable electronic device to control a playing mode of a headphone connected to the portable electronic device. The headphone controlling system includes a processor, a storage module, a manual controlling module, a positioning module, and a switching module. The storage module stores map data, street traffic data, and automatic control information. The manual controlling module manually switches the playing mode; when it is determined that the portable electronic device is on a heavy traffic street, the positioning module transmits the automatic control information in the storage module to the processor to generate and transmit a control signal according to the automatic control information to the switching module, thereby automatically switching the playing mode. A portable electronic device employing the headphone controlling system is also provided.