A vehicle remote instruction system that includes a remote autonomous driving vehicle and a remote instruction apparatus in which the remote commander issues the remote instruction includes: a remote instruction request reception unit configured to receive the remote instruction request; a position information acquisition unit configured to acquire position information on a location where the remote autonomous driving vehicle transmitted the remote instruction request; a request occurring location determination unit configured to determine a location where the remote instruction request is transmitted at equal to or higher frequency than an instruction request frequency set in advance, based on the acquired remote instruction request and the position information; and a passing-detour location setting unit configured to set the location where the remote instruction request is transmitted at equal to or higher frequency than the instruction request frequency, as a passing-detour location for the remote autonomous driving vehicle.