Systems, methods, and software described herein manage descriptive attributes associated with objects in video data. In one example, a video processing service obtains video data and identifies an object of interest in the video data. The video processing service further identifies attributes for the object of interest from the video data based on the attributes satisfying uniqueness requirements to differentiate the object of interest from other objects. Once the attributes are identified, the attributes may be stored in a storage system to be compared against other objects in second video data.