A system comprises compute nodes distributed over a network and configured to perform a pipeline parallel process. The system also comprises an extended memory comprising a global virtual address space which is shared by the compute nodes. The extended memory is configured to enable the compute nodes to exchange data over the network when the compute nodes perform the pipeline parallel process.