A reference voltage controlling circuit and an analog-to-digital converter are disclosed. The reference voltage controlling circuit includes a reference voltage generating circuit, a plurality of groups of sampling switching units and a logic controlling circuit. The DAC capacitor array switches the sampling switching units to a second positive reference voltage and a second negative reference voltage before starting sampling or conversion, and is charged and discharged with the second positive reference voltage and the second negative reference voltage to raise a voltage to a preset voltage. The sampling switching unit is switched to a first positive reference voltage and a first negative reference voltage to charge and discharge the DAC capacitor array to a target voltage. The rising of the voltage from the preset voltage to the target voltage is completed by the first positive reference voltage and the first negative reference voltage.