A network may provide latency optimization by configuring respective latency values of one or more network components. A latency manager may receive a request indicative of a maximum latency value of a communications path between two devices, and may determine a particular network latency value. The latency manager may then determine respective endpoint latency values for endpoint devices based on the maximum latency value and network latency values. In addition, buffer characteristics, such as buffer depth at particular devices, may be adjusted in view of the latencies.