A device includes a gate isolation plug, which further includes a U-shaped layer having a bottom portion and two sidewall portions, and an inner region overlapping the bottom portion. The inner region contacts the two sidewall portions. A first transistor has a first gate stack, and a first end of the first gate stack is in contact with both the inner region and the U-shaped layer of the gate isolation plug. A second transistor has a second gate stack, and a second end of the second gate stack is in contact with both the inner region and the U-shaped layer of the gate isolation plug. The first gate stack and the second gate stack are on opposite sides of the gate isolation plug.