A storage system facilitates deduping repeating data segments when generating a Manifest hierarchy for a file. During operation, the system can select a partitioning function that identifies a pattern that is expected to occur a predetermined number of times within the file. The system can process a plurality of segments of the file, using the partitioning function, to identify a set of chunk boundaries. The system generates a chunk for each file portion between two consecutive chunk boundaries, and generates a Manifest that includes a Content Object Hash (COH) value for each partitioned chunk. The system can store the Manifest and the unique partitioned chunks in a storage repository, such that two partitioned chunks with a common COH value are stored once in the storage repository.