A method for performing question answer (QA) tasks that includes entering an input into an encoder portion of an adaptive memory network, wherein the encoder portion parses the input into entities of text for arrangement of memory banks. A bank controller of the adaptive memory network organizes the entities into progressively weighted banks within the arrangement of memory banks. The arrangement of memory banks may be arranged to have an initial memory bank having lowest relevance for lowest relevance entities being closest to the encoder, and a final memory bank having a highest relevance for highest relevance entities being closes to a decoder. The method may continue with inferring an answer for the question answer (QA) task with the decoder analyzing only the highest relevance entities in the final memory bank.