A retractable handle system for a vehicle may include an electric actuator disposed inside a door panel, a driving member configured to be rotated in forward and reverse directions by receiving rotational force of the electric actuator, a driving arm provided to be rotated in the forward and reverse directions by the rotation of the driving member and blocked by the driving member from being rotated in a state in which an operation of the electric actuator is stopped so as to maintain a stored state and a pop-up state of a handle, and the handle which is supported by and coupled to the driving arm and configured to be moved by the forward and reverse rotation of the driving arm.