Disclosed therein are system and method for controlling a virtual network with a security function which can manage security states of virtual machines in a cloud datacenter, analyze security states of malicious virtual machines, and isolate and treat the malicious virtual machines in order to cope with intrusion of a virtual network under a cloud computing environment. The virtual network controlling system and method reduce the number of packets to which the IPS carries out a signature matching inspection through a DPI test by diffusing blocking against the previously detected intruder by the network level, so as to enhance performance of the virtualized network IPS.