An embodiment of the invention sets forth a primary processing unit, a secondary processing unit coupled to the primary processing unit and accessible via a plurality of channels and a plurality of guest virtual machines executing on the primary processing unit. Each guest virtual machine includes a driver associated with the secondary processing unit, and a privileged virtual machine executing on the primary processing unit and configured to allocate a different set of channels of the plurality of channels to each of the drivers included in the guest virtual machines, where a first set of channels allocated to a first driver enables the first driver to access the secondary processing unit without conflicting with any of the other and with minimal performance overhead by directly accessing the secondary processing unit channels.