Disclosed is a system and a method for geo-referencing an object on a floor. The method comprises receiving a camera frame providing view of a section of the floor, receiving a detailed floor plan of the floor, selecting a first area from the section of the floor in which the object needs to be tracked and defining a plurality of reference points. Herein, at least one of the pluralities of reference points is close to a mid-point along one of corresponding axes of the first area. The method further comprises mapping the plurality of reference points to the floor plan, processing the camera frame to determine if the object has been placed in the first area and geo-referencing the one or more object in the floor plan if the object is determined to be placed in the first area.