Embodiments of the invention enable application programs running across multiple compute nodes of a highly-parallel system to selectively migrate objects from one node to another. For example, when an object becomes too large, a node containing the object may migrate the object to another node, thereby freeing memory space. Whether a large object is migrated can be dependent on how frequently the object is used by the application. Because the memory used by such an object is freed for other uses by the application, overall application performance may be improved. On large parallel systems with thousands of compute nodes, even relatively small improvements in application performance an individual compute node may be magnified many times, resulting in dramatic improvements in overall application performance.