An on-chip accelerator manager manages multiple accelerators in a programmable device. In one specific implementation, the multiple accelerators are identical accelerators. The accelerator manager and the multiple accelerators are deployed to the programmable device. One or more calls to a software library in a virtual function table are replaced with one or more calls to the on-chip accelerator manager. The on-chip accelerator manager receives an accelerator call, and in response, allocates the accelerator call to one of the multiple accelerators, performs load balancing to the multiple accelerators, and performs failover when one of the accelerators fails by allocating work that was allocated to the failed accelerator to a different accelerator. The on-chip accelerator can push work to the multiple accelerators in a first mode of operation, and the multiple accelerators can pull work from the on-chip accelerator manager in a second mode of operation.