A method and system to help manage latency in setup of fallback communications, such as circuit switched fallback calls for instance, by taking into account the load on various fallback carriers. When a first network is serving a device and engages in a process to facilitate setup of a communication for the device to be served by a second, fallback network, the first network will work with the second network to determine which local carriers of the second network are least loaded, and the first network will then direct the WCD to scan those determined least-loaded carriers in particular. This process may thus help to avoid having the WCD spend time scanning local carriers of the second network that the second network is unlikely to assign in any event due to loading.