Methods and systems for detecting abnormal application behavior include determining a vector representation of a first syscall graph that is generated by a first application, the vector representation including a representation of a distribution of subgraphs of the first syscall graph. The vector representation of the first syscall graph is compared to one or more second syscall graphs that are generated by respective second applications to determine respective similarity scores. It is determined that the first application is behaving abnormally based on the similarity scores, and a security action is performed responsive to the determination that the first application is behaving abnormally.