System and method for fast parallel data processing in distributed storage systems
Abstract:
A system and method is disclosed for fast parallel data processing in a distributed storage system. An example method includes receiving at least one digital object; determining, by a processor of a computing device, whether the at least one digital object has a fixed data block structure; in response to determining that the at least one digital object has a fixed data block structure, determining by the processor a size of each fixed data block of the at least one digital object; determining a number of archive chunks for storing the at least one digital object on a selected plurality of storage servers based at least upon the size of each fixed data block; and partitioning the at least one digital object into the number of archive chunks for storing on the selected plurality of storage servers using erasure coding.
Information query
Patent Agency Ranking
0/0