A multi-layer video system has a first layer encoder that encodes a first layer of video information, at least one second layer encoder that encodes at least one second layer of video information, and an encoder side reference processing unit (RPU) that estimates one or more of an optimal filter or an optimal process that applies on a reference picture that is reconstructed from the first video information layer, and processes a current picture of the second video information layer, based on a correlation between the first layer reconstructed reference picture. The correlation relates to a complexity characteristic that scaleably corresponds to the first video information layer reconstructed reference picture and the second video information layer current picture. A scalable video bitstream is outputted, which may be decoded by a compatible decoder. A decoder side RPU and the encoder side RPU function as an RPU pair.