A multi-color off-axis digital holographic system and the imaging method thereof are disclosed. The multi-color off-axis digital holographic system comprises: a plurality of light emitting diodes, for provide a red (R) beam, a green (G) beam and a blue (B) beam; an interference object lens module, for receiving the R, G, and B beams to generate a beam containing an interference signal; a color imaging device, for receiving the beam containing the interference signal and thus forming a hologram on a surface of the color imaging device by holographic interference while registering the hologram; and a processing device, for receiving the registered hologram form the color imaging device; wherein the processing device perform a zero-filling and reconstructing operations upon the received hologram to obtain phase information of the R, G and B beams. With the aforesaid system, a three-dimensional surface profile with respect to a RG synthetic wavelength is obtained according to a calculation using the phase information of the R and G beams as well as the wavelengths thereof, and similarly, a three-dimensional surface profile with respect to a GB synthetic wavelength is obtained according to a calculation using the phase information of the G and B beams as well as the wavelengths thereof. Thereafter, by performing a calculation using the RG synthetic wavelength and its phase as well as the GB synthetic wavelength and its phase, an overall three-dimensional surface profile with respect to a complete synthetic wavelength is obtained.