An apparatus for controlling a transmission system of a hybrid electric vehicle may include: a planetary gear set disposed on an input shaft; two input gears connected to rotation elements of a sun gear, a ring gear and a planet carrier of the planetary gear set; first and second motor/generators disposed on an intermediate shaft and a connecting shaft disposed in parallel with the input shaft; two intermediate gears connected respectively to a rotor of the first and second motor/generators; a brake selectively braking the rotor of the first motor/generator; two output gears; a hydraulic pump; and a controller. The controller performs a hydraulic pressure control for the brake through the hydraulic pump to lock up the brake when a conversion condition is satisfied, and a reaction force control for the sun gear when a driving mode is converted from a power split mode to an engine coupling mode.