A method for detection of malicious files includes training a mapping model for mapping files in a probability space. A plurality of characteristics of an analyzed file is determined based on a set of rules. A mapping of the analyzed file in probability space is generated based on the determined plurality of characteristics. A first database is searched using the generated mapping of the analyzed file to determine whether the analyzed file is associated with a family of malicious files. The first database stores mappings associated with one or more families of malicious files. In response to determining that the analyzed file is associated with the family of malicious files, a selection of one or more methods of malware detection is made from a second database. The second database stores a plurality of malware detection methods. The selected method is used to detect the associated family.