Exemplary embodiments of the invention provide a system and method for providing multicast service in a unicast-based Voice over Internet Protocol (VoIP) system. The system and method include a VoIP call server receiving subnet information from VoIP terminals authenticated by an authentication server, generating multicast group information, and providing the multicast group information to the respective authenticated VoIP terminals. The VoIP call server searches a list of grouped VoIP terminals, selects a VoIP terminal of a corresponding subnet, transmits multicast session information to the selected VoIP terminal to set the VoIP terminal as a relay VoIP terminal, and requests a media server to transmit group message data to the relay VoIP terminal. According to this system and method, it is possible to implement a multicast service in a unicast-based VoIP system without adding additional equipment.