Techniques and tools are described for encoding animation video. In some embodiments, a video encoder designates animation video for encoding as animation content, which typically involves changing one or more encoder settings or rules to improve encoding performance for the animation content. When the encoder encodes the animation video, the encoder detects edges in the animation video using texture and changes settings for areas that include detected edges so as to improve encoding quality for the areas. In some embodiments, a video encoder adjusts differential quantization rules and quantizes animation video according to the adjusted differential quantization rules.