Invention Grant
- Patent Title: Systems and methods for spark lineage data capture
-
Application No.: US17225886Application Date: 2021-04-08
-
Publication No.: US11681721B2Publication Date: 2023-06-20
- Inventor: Shalu Chadha , Ravi Kumar Sanjeevi , Sarath Chandra Bhargav Jiguru , Madhu Kotagiri , Nikesh Bisen , Ramana Chelkala , Rajesh Dadi
- Applicant: JPMORGAN CHASE BANK, N.A.
- Applicant Address: US NY New York
- Assignee: JPMORGAN CHASE BANK, N.A.
- Current Assignee: JPMORGAN CHASE BANK, N.A.
- Current Assignee Address: US NY New York
- Agency: Greenberg Traurig LLP
- Priority: IN 2011019617 2020.05.08
- Main IPC: G06F16/248
- IPC: G06F16/248 ; G06F16/25 ; G06F16/242 ; G06F16/901

Abstract:
Systems and methods for SPARK lineage data capture are disclosed. In one embodiment, in an information processing apparatus comprising at least one computer processor, a method for lineage data capture may include: (1) receiving, at a lineage engine and from a listener service, a decisive logical plan for a job; (2) extracting, using a plan parser, lineage data from the decisive logical plan; (3) producing, by a job lineage builder, job lineage data and job attribute data from the lineage data; (4) extracting, by the job lineage builder and from the job lineage data and the job attribute data, attribute information, transformation information, and estimate information for the job; and (5) storing, in a database, the attribute information, the transformation information, and the estimate information.
Public/Granted literature
- US20210349910A1 SYSTEMS AND METHODS FOR SPARK LINEAGE DATA CAPTURE Public/Granted day:2021-11-11
Information query