A method to build a 3D model for a physical object includes the following steps. First, depth information of a plurality of images is converted into a plurality of 3D point clouds. Then, the motion parameters of the camera to gather each image relative to a previous image are determined according to the 3D point clouds. Next, a registration energy estimate value of each image's 3D point cloud, as being integrated into a previous image's 3D point cloud, is determined according to corresponding motion parameters. Then, the motion parameters are varied to minimize the estimate value. An optimal camera path is determined according to the varied motion parameters. Finally, a 3D model of the physical object is built according the optimal camera path.