An audio identification system determines whether a portion of a sample of an audio signal includes silence and generates a test audio fingerprint for the audio signal based on the presence of silence. In one embodiment, the audio identification system uses a value indicating silence for a portion of the test audio fingerprint corresponding to the portion of the audio signal that includes silence. When comparing the test audio fingerprint to reference audio fingerprints, the portion of the test audio fingerprint including the value indicating the presence of silence is not used. In another embodiment, the audio identification system replaces the portion including silence with additive audio and generates a test audio fingerprint for comparison based on the resulting modified sample.