A rolling bearing device includes a rotary shaft; a housing; a pair of rolling bearings; an outer ring spacer; an elastic member provided in a bottom side of the housing, the bottom side of the housing being a second axial side of the housing, and the elastic member being configured to urge the rolling bearings, the outer ring spacer, and the rotary shaft toward a first axial side; and a retaining ring that is fitted into a circumferential groove provided on an inner periphery in the first axial side of the housing, the retaining ring being in axial contact with the outer ring of the rolling bearing located on the first axial side, and the retaining ring including at least three arc members arranged in a circumferential direction.