An adaptive security policy based scalable video service apparatus includes a video streaming server, an adaptive security policy server and a terminal. The video streaming server receives a service demand via a network and generates an encrypted streaming data. The adaptive security policy server analyzes a media structure and the service demand, by using a service profile received from the video streaming server, so as to generate a security policy description. The terminal generates and transmits the service demand to the video streaming server or the adaptive security server, obtains the encrypted streaming data from the video streaming server and decrypts the encrypted streaming data for playback, storing and retransmission.