An improved solution for categorizing moving objects into familiar colors in video is provided. In an embodiment of the invention, a method for categorizing moving objects into familiar colors in video comprises: receiving a video input; determining at least one object track of the video input; creating a normalized cumulative histogram of the at least one object track; and one of: performing a parameterization quantization of the histogram including separating the histogram into regions based on at least one surface curve derived from one of saturation and intensity; or identifying a significant color of the quantized histogram.