Described is an approach to enabling gesture interactions for the viewport widget in a graphical user interface (GUI) library. The gesture interactions may include continuous operations such as panning, zooming and rotating of the viewport's content with fingers (or styluses). The approach is based on using a camera to detect occlusion patterns in a sensor grid rendered over the viewport. The sensor grid consists of sensor blobs, which are small blobs of pixels with a distinct color. A sensor blob is aware of its location in both the viewport's coordinate system and the camera's coordinate system, and triggers an occlusion event at the location when it is occluded by a finger (or stylus). Robust techniques are devised to eliminate unintentional gestures, provide visual guidance and feedback for interactions, and minimize the visual interference of the sensor grid with the viewport's content.