Various embodiments concern video patient monitoring with detection zones. Various embodiments can comprise a camera, a user interface, and a computing system. The computing system can be configured to perform various steps based on reception of a frame from the camera, including: calculate a background luminance of the frame; monitor for a luminance change of a zone as compared to one or more previous frames, the luminance change indicative of patient motion in the zone; and compare the background luminance to an aggregate background luminance, the aggregate background luminance based on the plurality of frames. If the background luminance changed by more than a predetermined amount, then the aggregate background luminance can be set to the background luminance, luminance information of the previous frames can be disregarded, and motion detection can be disregarded.