In accordance with disclosed embodiments, there are provided systems, methods, and apparatuses for implementing a scalable scheduler with heterogeneous resource allocation of large competing workloads types using Quality of Service (QoS) requirements. For instance, according to one embodiment, there is disclosed a system to implement a scheduling service, in which the system includes: a processor and a memory to execute instructions at the system; a local cache allocated within the memory of the system; a compute resource discovery engine to identify a plurality of computing resources available to execute workload tasks, the computing resources residing within any one of private or public datacenters or third party computing clouds and a plurality of resource characteristics for each of the plurality of computing resources identified; in which the compute resource discovery engine is to fill the local cache with information representing each of the identified computing resources available and the plurality of resource characteristics identified for each of the plurality of computing resources; a workload discovery engine to identify pending workload tasks to be scheduled for execution from one or more workload queues and to update the local cache with the identified workload tasks; a policy engine to identify a Service Level Target (SLT) for each of the workload tasks identified and to update the local cache with the SLT for each workload task identified; and a scheduler to schedule each workload task for execution via one of the computing resources available based on which of the computing resources are estimated to meet the SLT. Other related embodiments are disclosed.