Multi-piece golf balls containing a dual-layered core and dual-layered cover are provided. The core structure includes a small, heavy inner core (center) having a relatively high specific gravity, and a surrounding outer core layer. The core layers may have different hardness gradients. The center of the core comprises a metal material such as copper, steel, brass, tungsten, titanium, aluminum, and combinations and alloys thereof preferably dispersed in a thermoplastic polymeric matrix. The outer core layer is preferably formed from a thermoset composition such as polybutadiene. The multi-layered cores and covers have selective specific gravity relationships between the different layers. This helps provide the ball with good flight distance and spin control. The resulting balls have good resiliency and other playing performance properties.