An approach is provided for detecting a malicious activity on a computer system. First process trees are identified for computer processes that have been executed on a computer system. Each of the first process trees are vectorized. The vectorized first process trees are associated with respective labels. Each label represents an amount by which a respective vectorized process tree reflects the malicious activity. An artificial neural network is trained by using the vectorized first process trees and the associated labels as training input. After the training of the artificial neural network is completed, second process trees for currently executing computer processes are vectorized and provided as input vectors to the artificial neural network. Responsive to the artificial neural network providing an output indicating that a combination of the input vectors indicates the malicious activity, a remedial action is performed.