A method includes forming a semiconductor fin protruding higher than top surfaces of isolation regions. The isolation regions extend into a semiconductor substrate. The method further includes etching a portion of the semiconductor fin to form a trench, filling the trench with a first dielectric material, wherein the first dielectric material has a first bandgap, and performing a recessing process to recess the first dielectric material. A recess is formed between opposing portions of the isolation regions. The recess is filled with a second dielectric material. The first dielectric material and the second dielectric material in combination form an additional isolation region. The second dielectric material has a second bandgap smaller than the first bandgap.