Fast visual data annotation includes automatic detection using an automatic detector to detect subjects and joints in video frames. Then, annotation with sampling is performed, including determining when a frame is a sample (e.g., based on comparison of frames). Replay and refinement is utilized where user is involved with manually annotating subjects and/or joints in only select video frames.