The invention discloses a low-latency multi-driver adaptive noise canceling (ANC) system for a personal audio device. A personal audio device including multiple output transducers for reproducing different frequency bands of a source audio signal, includes an adaptive noise canceling (ANC) circuit that adaptively generates an anti-noise signal for each of the transducers from at least one microphone signal that measures the ambient audio to generate anti-noise signals. The anti-noise signals are generated by separate adaptive filters such that the anti-noise signals cause substantial cancelation of the ambient audio at their corresponding transducers. The use of separate adaptive filters provides low-latency operation, since a crossover is not needed to split the anti-noise into the appropriate frequency bands. The adaptive filters can be implemented or biased to generate anti-noise only in the frequency band corresponding to the particular adaptive filter. The anti-noise signals are combined with source audio of the appropriate frequency band to provide outputs for the corresponding transducers.