A semiconductor device with an isolation structure and a method of fabricating the same are disclosed. The semiconductor device includes first and second fin structures disposed on a substrate and first and second pairs of gate structures disposed on the first and second fin structures. The first end surfaces of the first pair of gate structures face second end surfaces of the second pair of gate structure. The first and second end surfaces of the first and second pair of gate structures are in physical contact with first and second sidewalls of the isolation structure, respectively. The semiconductor device further includes an isolation structure interposed between the first and second pairs of gate structures. An aspect ratio of the isolation structure is smaller than a combined aspect ratio of the first pair of gate structures.