The present disclosure relates to a communication method and system for converging a 5th-Generation (5G) communication system for supporting higher data rates beyond a 4th-Generation (4G) system with a technology for Internet of Things (IoT). The present disclosure may be applied to intelligent services based on the 5G communication technology and the IoT-related technology, such as smart home, smart building, smart city, smart car, connected car, health care, digital education, smart retail, security and safety services. Embodiments herein provide a method for authentication by dynamically generating security credentials in plug and play scenarios without a pre-configuration of F1 security credentials at an integrated access and backhaul (IAB) relay device in a wireless network. The method includes generating, by the IAB relay device, a stratum security key for one of an access stratums (AS) security establishment and a non-access stratums (NAS) security establishment with an IAB donor device in the wireless network. Further, the method includes generating, by the IAB relay device, a pre-shared key (PSK) based on the stratum security key. Further, the method includes generating an Internet Key Exchange (IKE) value using the PSK for establishing an F1 interface security with the IAB donor device.