A method of controlling a transmission of a vehicle, includes determining, by a controller, whether a shift prohibition condition for preventing overheating of the transmission of the vehicle in a terrain mode, which is a rough road driving mode of the vehicle, is satisfied; determining, by the controller, whether a shift operation of the transmission corresponding to a busy shift phenomenon of the transmission is detected when the shift prohibition condition is satisfied; and prohibiting, by the controller, shifting of the transmission when the shift operation of the transmission corresponding to the busy shift phenomenon of the transmission is detected.