The present invention relates to a method and apparatus for controlling a handshake operation. Datagram Transport Layer Security (DTLS) is an important secure protocol in the IP based Internet of things. The performance of DTLS handshake can be significantly affected by network status, traffic and packet loss rate, etc. It is therefore suggested evaluating a package loss rate and estimating causes of packet loss. Then, a DTLS handshake strategy may be changed adaptively based on the detection of packet loss and network status. As a result, the successful rate and delay of DTLS handshake can be improved. An acknowledgement and a non-acknowledgement mode may be used in a hybrid way to evaluate the package loss rate and estimate causes of packet loss and eventually improve performance of DTLS handshake.