Instructions of an application program are emulated such that they are carried out sequentially in a first virtual execution environment that represents the user-mode data processing of the operating system. A system API call requesting execution of a user-mode system function is detected. In response, the instructions of the user-mode system function called by the API are emulated according to a second emulation mode in which the instructions of the user-mode system function are carried out sequentially in a second virtual execution environment that represents the user-mode data processing of the operating system, including tracking certain processor and memory states affected by the instructions of the user-mode system function. Results of the emulating of the application program instructions according to the first emulation mode are analyzed for any presence of malicious code.