Embodiments are provided to enable graphic processing unit (GPU) virtualization for high bandwidth or rate demanding applications, such as 3D gaming, where a client communicates with a host via a virtual desktop infrastructure (VDI). The distributed GPU virtualization allows one or more VMs or comparable hosts or components without GPU access to communicate with a GPU at a different component or physical machine in a data center or a network using remote direct memory access (RDMA). A first physical machine that excludes a GPU starts a remote display driver function to handle a request to render graphics from a client via gateway. A second physical machine that comprises a GPU is instructed to start a render function for the client using the GPU. The render function communicates with the remote display driver function at the first physical machine. The rendered graphics is then sent to the client via the gateway.