A heterogeneous sensor fusion apparatus includes a point processor configured to detect a first object by processing a detection point input from a first sensor, an image processor configured to detect a second object by processing an image input from a second sensor, a point-matching unit configured to calculate a matching rate by matching the detection point with the image, and to determine whether the first object and the second object are identical based on the calculated matching rate, an association unit configured to generate track information by fusing information from the first sensor and the second sensor when the first object and the second object are identical, and an output unit configured to output the generated track information.