A coated article includes a silver (Ag) based infrared (IR) reflecting layer(s) on a glass substrate that is provided adjacent to and contacting at least one metallic or substantially metallic zinc (Zn) inclusive barrier layer in order to improve chemical durability characteristics of the low-E coating. In certain example embodiments, the silver based layer may be sandwiched between first and second metallic or substantially metallic barrier layers of or including zinc. The IR reflecting layer(s) and zinc based barrier layer(s) are part of a low emissivity (low-E) coating.