An apparatus and method are described for designing an accelerator for processing sparse data. For example, one embodiment comprises a machine-readable medium having program code stored thereon which, when executed by a processor, causes the processor to perform the operations of: analyzing input graph program code and parameters associated with a target accelerator in view of an accelerator architecture template; responsively mapping the parameters onto the architecture template to implement customizations to the accelerator architecture template; and generating a hardware description representation of the target accelerator based on the determined mapping of the parameters to apply to the accelerator architecture template.