A method and apparatus for deep content tagging. A media device receives one or more first frames of a content item, where the one or more first frames spans a duration of a scene in the content item. The media device detects one or more objects or features in each of the first frames using a neural network model and identifies one or more first genres associated with the first frames based at least in part on the detected objects or features in each of the first frames. The media device further controls playback of the content item based at least in part on the identified first genres.