An audio device and a method of operating the same are provided. The audio device includes a storage unit, a first memory and a second memory, a hardware decoder, a software decoder, a first direct memory access (DMA) block, a second DMA block, and a controller. The controller converts the audio device from an ultra low power mode in which the first PCM information is transmitted to an audio interface buffer through the first memory, the hardware decoder, and the first DMA block or a low power mode in which the second PCM information is transmitted to the audio interface buffer through the second memory, the software decoder, and the first DMA block to a normal mode in which the second PCM information is transmitted to the audio interface buffer through the second memory, the software decoder, and the second DMA block.