Object annotation is images is tedious time consuming task when large volume of data needs to annotated. Existing methods limit to semiautomatic approaches for annotation. The embodiments herein provide a method and system for a deep network based architecture for automatic object annotation. The deep network utilized is a two stage network with first stage as an annotation model comprising a Faster Region-based Fully Convolutional Networks (F-RCNN) and Region-based Fully Convolutional Networks (RFCN) providing for two class classification to generate annotated images from a set of single object test images. Further, the newly annotated test object images are then used to synthetically generate cluttered images and their corresponding annotations, which are used to train the second stage of the deep network comprising the multi-class object detection/classification model designed using the F-RCNN and the RFCN as base networks to automatically annotate input test image in real time.