A composite masking system and method for improving the invisibility of high-definition video watermarking. The composite masking system includes a watermark generation module, a mask generation module, and watermark embedment means. The watermark generation module generates a basic watermark pattern using a private key, and generate a watermark pattern by repeatedly extending the basic watermark pattern. The mask generation module generates a Noise Visibility Function (NVF) mask using NVF masking means, an adaptive dithering mask using adaptive dithering masking means, and a contour mask using contour masking means. The watermark embedment means generates a composite mask by multiplying the NVF mask, the adaptive dithering mask and the contour mask together, multiplying the composite mask and the extended watermark pattern together, and embedding the result of the second multiplication in the luminance channel of an original image.