A system and method for operator-side privacy zone masking of surveillance is provided. The system includes a video surveillance camera equipped with a coordinate engine for determining coordinates of a current field of view of the surveillance camera; and a frame encoder for embedding the determined coordinates with video frames of the current field of view. The system also includes a privacy zone information database for storing coordinates of predefined privacy zones; and a video viewing unit for viewing stored video frames. The video viewing unit is in communication with the frame storage unit and the privacy zone information database.