Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for overlaying content on video streams. In one aspect, a category of a video is identified. Confidence scores are determined that each indicate a likelihood that a location in a frame includes a feature of a feature type. A weight for each feature type is determined, based on the category, that reflects an importance of not occluding features of the feature type. Confidence scores are adjusted, for each feature type, based on the weight for the feature type, to generate adjusted confidence scores. The adjusted confidence scores are aggregated for each location for each frame to generate aggregated and adjusted confidence scores. A location at which to position overlaid content during video display is determined based on the aggregated and adjusted confidence scores. The overlaid content is provided for display at the determined location in the video.