System and method for automatically obtaining data lineage in real time
Abstract:
A system includes an execution engine and a processor. The execution engine receives application code that, when executed, is configured to generate a second data element from a first data element that is stored in a first database and store the second element in a second database. The execution engine converts the code into an execution plan and executes it. The execution plan includes a first operation to obtain the first element from the first database, a second operation to apply a transformation to the first element, and a third operation to store the first element as the second element in the second database. The processor accesses the execution plan and determines that the third operation includes storing the second element. In response, the processor generates a data lineage for the second element by extracting, from each operation from the third to the first, a portion of the data lineage.
Information query
Patent Agency Ranking
0/0