An image sensor has an array of pixels of different colors. The pixels may be arranged in a repeating pattern of eight pixels having four rows and two columns. During charge summing operations, the first and third rows may share a floating diffusion and the second and fourth rows may share a floating diffusion. When charge summing is inactive, transfer gates in the first and second columns may be controlled independently, while transfer gates in pairs of rows may be controlled simultaneously. When charge summing is active, summed charges from pixels of the same color in the first and third rows may be placed on the floating diffusion shared by the first and third rows and summed charges from pixels of the same color in the second and fourth rows may be placed on the floating diffusion shared by the second and fourth rows.