Methods, techniques, and systems for user interface remoting using video streaming techniques are provided. Example embodiments provide User Interface Remoting and Optimization System (“UIROS”), which enables the efficient remoting of user interfaces on behalf of their guests using generic video streaming techniques, such as H.264, to send compressed user interface image information in the form of video frame encoded bitstreams. With application cooperation, pixels are explicitly cached on the server using lookahead methods to lower latency in remoting the user interface for certain operations. In one embodiment, the UIROS comprises server side support including a UI remoting server, a video encoder, and rendering support and client side support including a UI remoting client, a video decoder, and a display. These components cooperate to implement optimized UI remoting that is bandwidth efficient, low latency and CPU efficient.