A method of forming an integrated circuit structure includes providing a semiconductor substrate; forming patterned features over the semiconductor substrate, wherein gaps are formed between the patterned features; filling the gaps with a first filling material, wherein the first filling material has a first top surface higher than top surfaces of the patterned features; and performing a first planarization to lower the top surface of the first filling material, until the top surfaces of the patterned features are exposed. The method further includes depositing a second filling material, wherein the second filling material has a second top surface higher than the top surfaces of the patterned features; and performing a second planarization to lower the top surface of the second filling material, until the top surfaces of the patterned features are exposed.