Remote desktop servers include a display encoder that maintains a secondary framebuffer that contains display data to be encoded and transmitted to a remote client display. The display encoder submits requests to update the display data in the secondary framebuffer to a video adapter driver that has access to a primary framebuffer whose display data is updated according to drawing commands received from applications running on the remote desktop servers. The video adapter driver utilizes a spatial data structure to track changes made to the display data located in regions of the primary framebuffer and copies the display data in those regions of the primary framebuffer to corresponding regions in the secondary framebuffer.