Embodiments herein provide a system (100) to estimate the amplitude of oscillations in a turbulent flow system (102) that exhibits oscillatory instabilities. The system (100) comprises of a sensor (102A) mounted on the turbulent flow system (102) to detect an oscillatory variable in the system obtaining a signal, a signal conditioner (104) that conditions the signal from the sensor, an amplitude estimator (110) that estimates the amplitude of the limit cycle oscillations, and also predict the proximity of the system to the oscillatory instability, a processor (108) connected to the amplitude estimator (110) to compare the predicted oscillation amplitude with a threshold value. The amplitude is estimated by estimating the spectral measure of the time series signal obtained from the system.