A system and a method for on-demand search of a large data-set is presented. The system includes a data indexer, an index writer, and an index reader. The data indexer is configured to index the data set. The index writer is configured to create a multi-level directory including a plurality of directories having one or more hash partitions. The index writer is further configured to generate a hash table for each directory and write data from the indexed data set into a corresponding hash partition of a directory. The index reader is configured to identify and query a hash partition in each directory based on a search term and a corresponding hash table for the directory. The index reader is further configured to retrieve one or more relevant records, and present the one or more relevant records to a user.