The system determines whether content such as an image is suitable for content modification based on one or more criteria. The system includes decision engines or modules configured to evaluate one or more suitability metrics based on corresponding criteria such as publication status, restriction status, context, compatibility, and classification. If content is unsuitable for content modification because of entities or context depicted therein, privacy status, incompatibility with content modification, properties of the content file itself, or other aspects, the system generates a tag indicating the content is unsuitable for content modification. If content is suitable for content modification because of entities or context depicted therein, publication status, compatibility with content modification, properties of the content file itself, or other aspects, the system generates a content modification tag indicating the content is suitable for content modification. Metadata associated with the content is updated based on the flag.