Provided are a computer program product, system, and method for selecting pages implementing leaf nodes and internal nodes of a data set index for reuse in memory. Pages in the memory are allocated to internal nodes and leaf nodes of a tree data structure representing all or a portion of a data set index for the data set. At least at least one duration parameter indicates an amount of time after which an unused page implement one of the internal nodes or leaf nodes is eligible for reuse. Selection is made of at least one selected page allocated to at least one of the leaf nodes and the internal nodes that has not been used for one of the at least one duration parameter for reuse.