An approach for improved security protocols in a mobile satellite system is provided. A remote terminal performs a key establishment function, including determination of a first encryption key for encrypting data for transmission over the satellite communications channels, and determination of an authentication key for authenticating entities communicating over the communications channels. The remote terminal receives a security mode command including a key indicator, and determines a second encryption key for enhanced session data security over communications channels. The second encryption key is determined based on the key indicator and a key generation algorithm. The remote terminal further determines a key indicator response and transmits a security mode complete command including the key indicator response to a satellite base station subsystem (SBSS). The key indicator response is constructed for the SBSS to determine the second encryption key based on the key indicator response and a key generation algorithm.