Abase station (BS) of handling communication with a communication device comprises a storage device for storing instructions and a processing circuit coupled to the storage device. The processing circuit is configured to execute the instructions stored in the storage device. The instructions comprise configuring a radio resource control (RRC) connection to the communication device; transmitting an RRCConnectionRelease message via the RRC connection to the communication device; determining that the communication device leaves a connected mode after a first duration from transmitting the RRCConnectionRelease message, when the communication device is in a coverage enhancement (CE) mode; and determining that the communication device leaves the connected mode after a second duration from transmitting the RRCConnectionRelease message, when the communication device is not in the CE mode.