A package includes a device die including a first plurality of metal pillars at a top surface of the device die. The package further includes a die stack including a plurality of dies bonded together, and a second plurality of metal pillars at a top surface of the die stack. One of the device die and the plurality of dies includes a semiconductor substrate and a through-via penetrating through the semiconductor substrate, A polymer region includes portions encircling the device die and the die stack, wherein a bottom surface of the polymer region is substantially level with a bottom surface of the device die and a bottom surface of the die stack. A top surface of the polymer region is level with top ends of the first and the second plurality of metal pillars. Redistribution lines are formed over the first and the second plurality of metal pillars.