Described is a system for neural decoding of neural activity. Using at least one neural feature extraction method, neural data that is correlated with a set of behavioral data is transformed into sparse neural representations. Semantic features are extracted from a set of semantic data. Using a combination of distinct classification modes, the set of semantic data is mapped to the sparse neural representations, and new input neural data can be interpreted.