Invention Grant
- Patent Title: Dataflow execution time estimation for in-memory distributed processing framework
-
Application No.: US16040774Application Date: 2018-07-20
-
Publication No.: US10901782B2Publication Date: 2021-01-26
- Inventor: Vinícius Michel Gottin , Jonas F. Dias , Edward José Pacheco Condori , Angelo E. M. Ciarlini , Bruno Carlos da Cunha Costa , Fábio André Machado Porto , Paulo de Figueiredo Pires , Yania Molina Souto , Wagner dos Santos Vieira
- Applicant: EMC IP Holding Company LLC
- Applicant Address: US MA Hopkinton
- Assignee: EMC IP Holding Company LLC
- Current Assignee: EMC IP Holding Company LLC
- Current Assignee Address: US MA Hopkinton
- Agency: Ryan, Mason & Lewis, LLP
- Main IPC: G06F9/48
- IPC: G06F9/48 ; G06F9/50

Abstract:
Techniques are provided for dataflow execution time estimation for distributed processing frameworks. An exemplary method comprises: obtaining an input dataset for a dataflow for execution; determining a substantially minimal data unit for a given operation of the dataflow processed by the given operation; estimating a number of rounds required to execute a number of data units in the input dataset using nodes assigned to execute the given operation; determining an execution time spent by the given operation to process one data unit; estimating the execution time for the given operation based on the execution time spent by the given operation to process one data unit and the number of rounds required to execute the number of data units in the input dataset; and executing the given operation with the input dataset. A persistent cost model is optionally employed to record the execution times of known dataflow operations.
Public/Granted literature
- US20200026550A1 Dataflow Execution Time Estimation for In-Memory Distributed Processing Framework Public/Granted day:2020-01-23
Information query