An accelerator manager monitors hardware accelerators that are called by one or more computer programs. A virtual function table includes multiple entries, where each entry correlates a call from a computer program to a corresponding call to either a software library or a hardware accelerator. A call by the computer program to a function in the virtual function table results in the call being routed to either the software library or to a hardware accelerator depending on the contents of the corresponding entry in the virtual function table. The accelerator manager, in response to a detected failure in an accelerator, replaces one or more calls in the virtual function table to the failed accelerator with calls to the software library. The accelerator manager can then retry the call that caused the accelerator to fail, which will then be executed by the software library.