Provided are a computer program product, system, and method for using different secure erase algorithms to erase chunks from a file associated with different security levels. A request is received to secure erase a file having a plurality of chunks stored in at least one storage device. A determination is made of a first secure erase algorithm to apply to a first chunk in the file in response to the request and of a second secure erase algorithm to apply to a second chunk in the file in response to the request. The first secure erase algorithm is applied to erase the first chunk and the second secure erase algorithm is applied to erase the second chunk. The first and second secure erase algorithms use different processes to erase the chunks to which they are applied.