Data processing method and apparatus
Abstract:
In a data processing method, a worker node in a distributed data processing system receives first data from an upstream worker node. The first data has been stored in a buffer of the upstream worker node. The worker node sends a first portion of the first data to a persistent storage device of the distributed data processing system for persistent backup, and performs computational processing on the first data to generate second data. Prior to completing performing computational processing on the first data, the worker node sends acknowledgement information to the upstream worker node to instruct the upstream node to delete the first data from the buffer of the upstream worker node. The worker node then sends the second data to a downstream worker node in the distributed data processing system for further processing by the downstream worker node.
Public/Granted literature
Information query
Patent Agency Ranking
0/0