A system and method of updating a key based on communication and security damage costs are provided. The key updating system for group communication between a plurality of group members, includes a key update cycle determining unit to determine a key update cycle based on a communication cost and a security damage cost, and a key updating unit to perform a key updating with respect to one or more group members of the group communication based on the determined key update cycle.