Invention Grant
- Patent Title: Efficient out of process reshuffle of streaming data
-
Application No.: US16426683Application Date: 2019-05-30
-
Publication No.: US11010171B2Publication Date: 2021-05-18
- Inventor: Alexander Alperovich , Zhong Chen , Boris Shulman
- Applicant: Microsoft Technology Licensing, LLC
- Applicant Address: US WA Redmond
- Assignee: Microsoft Technology Licensing, LLC
- Current Assignee: Microsoft Technology Licensing, LLC
- Current Assignee Address: US WA Redmond
- Agency: Fiala & Weaver P.L.L.C.
- Main IPC: G06F15/173
- IPC: G06F15/173 ; G06F9/38 ; G06F16/27 ; G06F16/25

Abstract:
Methods, systems, apparatuses, and computer program products are provided for processing a stream of data. A maximum temporal divergence is established for data flushed to a data store from a plurality of upstream partitions. Each of a plurality of data flushers, each corresponding to an upstream partition, may obtain an item of data from a data producer. Each data flusher may determine whether flushing the data to the data store would exceed the maximum temporal divergence. Based at least on determining that flushing the data to the data store would not exceed the maximum temporal divergence, the data may be flushed to the data store for ingestion by a downstream partition and a data structure (e.g., a ledger) may be updated to indicate a time associated with the most recent item of data flushed to the data store.
Public/Granted literature
- US20200379774A1 EFFICIENT OUT OF PROCESS RESHUFFLE OF STREAMING DATA Public/Granted day:2020-12-03
Information query