Provided are a computer program product, system, and method for classifying a digital speech sample of a call to determine routing for the call. A digital speech sample of a call agent is processed to categorize speech as having an agent speech attribute. Information is received on a caller speech attribute initiating a call to the call center. The caller speech attribute results from categorizing a caller digital speech sample of the caller as having the caller speech attribute. A call agent of call agents having a call agent speech attribute that matches the caller speech attribute is selected and the call is routed to the selected call agent to process the call.