A resist composition is provided comprising a polymer comprising recurring units (a) having an oxazolidinedione, thioxooxazolidinone, thiazolidinedione or thioxothiazolidinone structure and recurring unit (b1) having an acid labile group-substituted carboxyl group and/or recurring units (b2) having an acid labile group-substituted phenolic hydroxyl group. The resist composition suppresses acid diffusion, exhibits a high resolution, and forms a pattern of satisfactory profile with low edge roughness.