A method for detecting DoS attacks using an encrypted communication protocol includes estimating traffic telemetries of packets of at least ingress traffic passing over an insecure network that is directed to a protected entity by analyzing TCP headers of the packets, the packets using an encrypted version of a non-encrypted communication protocol, the packets being intended for the protected entity; providing at least one rate-based feature and at least one rate-invariant feature based on the estimated traffic telemetries, wherein the rate-based feature and the rate-invariant feature demonstrate a normal behavior of the traffic; and executing a mitigation action when a potential flood DoS attack using the encrypted communication protocol is detected by an evaluation of each of the at least one rate-based feature and the at least one rate-invariant feature with respect to respective baselines to determine whether the behavior of the ingress traffic indicates a potential flood DoS attack.