Invention Grant
- Patent Title: Atomic incremental load for map-reduce systems on append-only file systems
-
Application No.: US15198345Application Date: 2016-06-30
-
Publication No.: US10558615B2Publication Date: 2020-02-11
- Inventor: Sandeep Tata
- Applicant: INTERNATIONAL BUSINESS MACHINES CORPORATION
- Applicant Address: US NY Armonk
- Assignee: International Business Machines Corporation
- Current Assignee: International Business Machines Corporation
- Current Assignee Address: US NY Armonk
- Agency: Sherman IP LLP
- Agent Kenneth L. Sherman; Steven Laut
- Main IPC: G06F16/16
- IPC: G06F16/16 ; G06F16/18 ; G06F16/182

Abstract:
Augmenting data files in a repository of an append-only file system includes maintaining a companion metadata file for each corresponding data file in a map-reduce system using the append-only file system. Each companion metadata file tracks a logical end-of-file (EOF) for each data file. Global versioning of each companion metadata is maintained. A map-reduce append job is performed for a set of data files using a current global version number for the companion metadata file. The map-reduce job including multiple append tasks. For each successful append job, a logical EOF for each appended file is incremented to a new physical EOF. For each failed append task of the append job, a logical EOF is maintained for each failed append task by not incrementing the logical EOF for each failed append task.
Public/Granted literature
- US20160306799A1 ATOMIC INCREMENTAL LOAD FOR MAP-REDUCE SYSTEMS ON APPEND-ONLY FILE SYSTEMS Public/Granted day:2016-10-20
Information query