A remote user interface (RUI) system, an RUI server, an RUI terminal apparatus and a controlling method are provided. The controlling method of the RUI server includes receiving stream data from an external server in response to streaming service request information being received a requesting RUI terminal apparatus among a plurality of RUI terminal apparatuses, determining whether it is possible perform, in the RUI server, trans-coding of the received stream data, and performing, based on a result of the determining, at least one of trans-coding the stream data, and transmitting trans-coding process information regarding the received stream data to RUI terminal apparatuses that can perform trans-coding among the plurality of RUI terminal apparatuses, other than the requesting RUI terminal apparatus.