A method and security system for detecting and mitigating encrypted denial-of-service (DoS) attacks. The system includes a DoS defense (DoSD) module configured to detect an encrypted DoS attack in an inbound traffic by analyzing attributes only in the inbound traffic that relate to at least one of a network layer and an application layer, wherein the DoSD module is further configured to mitigate a detected encrypted attack, the inbound traffic originates at a client and is addressed to a protected server; and a cryptographic protocol engine (CPE) configured to establish a new encrypted session between the client and the security system, decrypt requests included in the inbound traffic, and send encrypted responses to the client over the new encrypted session between the client and the security system.