Embodiments of the invention receive videos and feedback data associated with the videos from a client device and adaptively encode the videos based on the feedback data. The encoded videos are suitable to be transmitted over a network and displayed on the client device. Embodiments of an encoding server adaptively changes resolution of a video on the fly or scale the video quality up or down based on the factors described by the feedback data, including network condition for transmitting the encoded video, network delay, encoder and decoder processing capacity and feedback from viewers of the decoded video. Furthermore, the encoding server adaptively encodes the video based on a combination of various factors described by the feedback data.