According to one embodiment, when a predicted trajectory is received, a set of one or more features are extracted from at least some of the trajectory points of the predicted trajectory. The predicted trajectory is predicted using a prediction method or algorithm based on perception data perceiving an object within a driving environment surrounding an autonomous driving vehicle (ADV). The extracted features are fed into a predetermined DNN model to generate a similarity score. The similarity score represents a difference or similarity between the predicted trajectory and a prior actual trajectory that was used to train the DNN model. The similarity score can be utilized to evaluate the prediction method that predicted the predicted trajectory.