A method of manufacturing a double row rolling bearing device includes: providing a collet chuck having radially expandable separated parts and a tapered cone for radially expanding the separated parts; inserting the radially contracted collet chuck into a through hole before the double row rolling bearing is caulked to the hub shaft; caulking the cylindrical end of the hub shaft, with the separated parts being in contact with the end by radially expanding the collet chuck to prevent radially inward plastic deformation of the end; and contracting the collet chuck and extracting the collet chuck out of the hub shaft after the caulking.