An analog-to-digital converter (ADC) includes reference charge cancellation features to at least partially offset a voltage distortion on a bypass capacitor of a reference buffer due to a voltage reference hit taken by a switched capacitor bank with which the bypass capacitor is connected. The charge cancellation may be configured in logic to be input signal dependent because different resolved bits or transitions between resolved bits may cause different amounts of voltage reference hits. By adjusting the bypass capacitor in response to each of at least some of the reference hits while resolving a word of bits, the reference voltage signal provided by the bypass capacitor undergoes far less settling, remaining more stable and linear for a more accurate reference voltage. Furthermore, a smaller capacitor may be used for the bypass capacitor, reducing power consumption and area on chip.