Information loss in speech to text conversion and Inability to preserve vocal emotion information without changing the artificial intelligence model infrastructure in a conventional speech to speech translation system are essential drawback of the conventional techniques. Embodiments of the invention provide direct speech to speech translation system is disclosed. Direct speech to speech translation system uses a one-tier approach, creating a unified-model for whole application. The single-model ecosystem takes in audio (mel spectrogram) as an input and gives out audio (mel spectrogram) as an output. This solves the bottleneck problem by not converting speech directly to text but having text as a byproduct of speech to speech translation, preserving phonetic information along the way. This model also uses pre-processing and post-processing scripts but only for the whole model. This model needs parallel audio samples in two languages. The training methodology involves augmenting or changing both sides of the audio equally.