Abstract:
A method of applying scheduling policies is provided. The method includes distributing user data throughout a plurality of storage nodes through erasure coding, wherein the plurality of storage nodes are housed within a single chassis coupling the storage nodes as a cluster. The method includes receiving operations relating to a non-volatile memory of one of the plurality of storage nodes into a plurality of operation queues. The method includes evaluating each of the operations in the plurality of operation queues as to benefit to the non-volatile solid-state storage according to a plurality of policies. For each channel of a plurality of channels coupling the operation queues to the non-volatile memory, the method includes iterating a selection and an execution of a next operation from the plurality of operation queues, with each next operation having a greater benefit than at least a subset of operations remaining in the operation queues.
Abstract:
A method of applying scheduling policies is provided. The method includes distributing user data throughout a plurality of storage nodes through erasure coding, wherein the plurality of storage nodes are housed within a single chassis coupling the storage nodes as a cluster. The method includes receiving operations relating to a non-volatile memory of one of the plurality of storage nodes into a plurality of operation queues. The method includes evaluating each of the operations in the plurality of operation queues as to benefit to the non-volatile solid-state storage according to a plurality of policies. For each channel of a plurality of channels coupling the operation queues to the non-volatile memory, the method includes iterating a selection and an execution of a next operation from the plurality of operation queues, with each next operation having a greater benefit than at least a subset of operations remaining in the operation queues.
Abstract:
A method of applying an address space to data storage in a non-volatile solid-state storage is provided. The method includes receiving a plurality of portions of user data for storage in the non-volatile solid-state storage and assigning to each successive one of the plurality of portions of user data one of a plurality of sequential, nonrepeating addresses of an address space. The address range of the address space exceeds a maximum number of addresses expected to be applied during a lifespan of the non-volatile solid-state storage. The method includes writing each of the plurality of portions of user data to the non-volatile solid-state storage such that each of the plurality of portions of user data is identified and locatable for reading via the one of the plurality of sequential, nonrepeating addresses of the address space.
Abstract:
A method for a transactional commit in a storage unit is provided. The method includes receiving a logical record from a storage node into a transaction engine of a storage unit of the storage node and writing the logical record into a data structure of the transaction engine. The method includes writing, to a command queue of the transaction engine, an indication to perform an atomic update using the logical record and transferring each portion of the logical record from the data structure of the transaction engine to non-persistent memory of the storage unit as a committed transaction. A storage unit for a storage system is also provided.
Abstract:
Enabling data portability between systems, including: submitting, to a cloud-based metadata store storing metadata describing data stored in a plurality of storage systems each coupled for data communications with a corresponding on-premises cloud infrastructure of a plurality of on-premises cloud infrastructures, a query; selecting a subset of the data responsive to the query and stored in at least one first storage system coupled to at least one first on-premises cloud infrastructure; and storing, from the at least one first storage system to a second storage system coupled to a second on-premises cloud infrastructure, the subset of the data responsive to the query.
Abstract:
A method for updates in a storage system is provided. The method includes writing identifiers, associated with data to be stored, to storage units of the storage system and writing trim records indicative of identifiers that are allowed to not exist in the storage system to the storage units. The method includes determining whether stored data corresponding to records of identifiers is valid based on the records of the identifiers and the trim records.
Abstract:
Nonsequential readahead based on data access patterns, the method comprising: determining a set of access patterns for stored content; determining, based on the set of access patterns, a list of storage locations for content expected to be used; and prefetching, based on the list of storage locations for content expected to be used, one or more data objects.
Abstract:
Storage layer data obfuscation, including: determining a subset of a dataset to obfuscate in accordance with a security policy; generating, based at least in part on the security policy, an obfuscated snapshot of the dataset that is representative of the dataset with the subset of the dataset obfuscated; and sending, to a target computer system, the obfuscated snapshot from which a restored version of the dataset includes the subset of the dataset obfuscated.
Abstract:
Data transformation offloading in an artificial intelligence infrastructure that includes one or more storage systems and one or more graphical processing unit (‘GPU’) servers, including: storing, within the storage system, a dataset; identifying, in dependence upon one or more machine learning models to be executed on the GPU servers, one or more transformations to apply to the dataset; and generating, by the storage system in dependence upon the one or more transformations, a transformed dataset.
Abstract:
A method for updates in a storage system is provided. The method includes writing identifiers, associated with data to be stored, to storage units of the storage system and writing trim records indicative of identifiers that are allowed to not exist in the storage system to the storage units. The method includes determining whether stored data corresponding to records of identifiers is valid based on the records of the identifiers and the trim records.