Systems and methods of tracking objects around an ego vehicle, using state estimation of the objects, where state estimation may include an estimate of a position, velocity, acceleration and bounding box size of the tracked object are disclosed. An object may be detected at one or more poses over time. For each pose, a first bounding box may be generated. The largest bounding box may be determined. For each pose, the first bounding box may be expanded to an expanded bounding box. The expanded bounding box may match the largest bounding box. For each pose, the center of the expanded bounding box may be determined. The object may be tracked according to the center of the expanded bounding box.