A semiconductor device structure is provided. The semiconductor device structure includes a substrate having a base and a fin over the base. The semiconductor device structure also includes a first nanostructure over the fin and a second nanostructure over the first nanostructure. The semiconductor device structure further includes a gate stack wrapping around an upper portion of the fin, the first nanostructure, and the second nanostructure. In addition, the semiconductor device structure includes a first inner spacer between the fin and the first nanostructure and a second inner spacer between the first nanostructure and the second nanostructure. The semiconductor device structure includes a first low dielectric constant structure in the first inner spacer and a second low dielectric constant structure in the second inner spacer. The first low dielectric constant structure is larger than the second low dielectric constant structure.