公开(公告)号：US20240256550A1

公开(公告)日：2024-08-01

申请号：US18162616

申请日：2023-01-31

Applicant: Databricks, Inc.

Inventor： Utkarsh Agarwal ,  Shoumik Palkar ,  Alexander Behm ,  Sriram Krishnamurthy

IPC: G06F16/2455 ,  G06F11/34 ,  G06F16/22

CPC classification number: G06F16/24558 ,  G06F11/3409 ,  G06F16/221

Abstract: Disclosed herein is a method, system, or non-transitory computer readable medium for evaluating a query on a columnar dataset comprising one or more dictionaries associated with columns in the dataset. The method includes receiving a request to perform a query comprising at least a operator and a request to return information about a value of interest in a columnar dataset stored on cloud storage. At least one column in the columnar dataset is based on a dictionary. The dictionary maps one or more values for a column to one or more respective identifiers. The method determines whether to perform dictionary filtering for the query by calculating a metric based on one or more factors. Responsive to the metric being below a threshold, which may be predetermined, the method performs the dictionary filtering.

公开(公告)号：US20240256549A1

公开(公告)日：2024-08-01

申请号：US18162607

申请日：2023-01-31

Applicant: Databricks, Inc.

Inventor： Utkarsh Agarwal ,  Shoumik Palkar ,  Alexander Behm ,  Sriram Krishnamurthy

IPC: G06F16/2455 ,  G06F11/34 ,  G06F16/22

CPC classification number: G06F16/24558 ,  G06F11/3409 ,  G06F16/221

Abstract: Disclosed herein is a method, system, or non-transitory computer readable medium for evaluating a query on a columnar dataset comprising one or more dictionaries associated with columns in the dataset. The method includes receiving a request to perform a query comprising at least an operator for a columnar dataset on cloud storage. At least one column in the dataset is based on a dictionary, and the dictionary maps one or more values for a column to one or more respective identifiers. The method evaluates the operator on one or more values of the dictionary to generate an updated dictionary comprising updated values. The method may decode the updated dictionary into an updated column comprising updated data values.

公开(公告)号：US20240202211A1

公开(公告)日：2024-06-20

申请号：US18219314

申请日：2023-07-07

Applicant: Databricks, Inc.

Inventor： Alexander Balikov ,  Tathagata Das ,  Karthikeyan Ramasamy

IPC: G06F16/27 ,  G06F16/2455

CPC classification number: G06F16/278 ,  G06F16/24568

Abstract: A data processing service performs a rebalancing process for rebalancing stateful tasks on a cluster computing system. In one instance, the method for rebalancing stateful tasks is performed such that the per-operator partitions are spread across available executors of a cluster of the cluster computing system with respect to one or more statistics of the tasks. In one instance, the method for rebalancing stateful tasks is also performed such that the total number of stateful tasks are balanced per executor as long as this rebalancing does not imbalance the per-operator placements. In this way, the processing of stateful tasks can be spread across multiple executors in a relatively uniform manner, even though there may be an upfront cost of breaking the local caching on an executor.

公开(公告)号：US11948084B1

公开(公告)日：2024-04-02

申请号：US18162291

申请日：2023-01-31

Applicant: Databricks, Inc.

Inventor： Sue Ann Hong ,  Shi Xin ,  Timothee Hunter ,  Ali Ghodsi

IPC: G06N3/08 ,  G06N3/04 ,  G06N3/063 ,  G06N5/02 ,  G06N5/022 ,  G06F16/14 ,  G06F16/22

CPC classification number: G06N3/08 ,  G06N3/04 ,  G06N3/063 ,  G06N5/022 ,  G06N5/027 ,  G06F16/14 ,  G06F16/22

Abstract: A function creation method is disclosed. The method comprises defining one or more database function inputs, defining cluster processing information, defining a deep learning model, and defining one or more database function outputs. A database function is created based at least in part on the one or more database function inputs, the cluster set-up information, the deep learning model, and the one or more database function outputs. In some embodiments, the database function enables a non-technical user to utilize deep learning models.

公开(公告)号：US20240070155A1

公开(公告)日：2024-02-29

申请号：US17895882

申请日：2022-08-25

Applicant: Databricks, Inc.

Inventor： Bart Samwel ,  Tathagata Das ,  Lars Kroll ,  Yijia Cui ,  Juliusz Sompolski ,  Tom Van Bussel

IPC: G06F16/2455 ,  G06F16/22

CPC classification number: G06F16/2456 ,  G06F16/2282

Abstract: A method, system, and computer system for performing an operation with respect to a target table are disclosed. The method includes performing first and second jobs, and obtaining other resulting files based at least in part on a second set of unmatched rows among the target table and the source table that results from the first set of unmatched rows having been processed in the second job, and obtaining a resulting table based on (i) second job resulting file(s), and (ii) other resulting files. Performing the first job includes determining a set of matching target table files and storing target table information indicating for each of the set of matching target table files, a particular set of rows having matching rows. Performing the second job includes performing a first matching action based on matched rows and a second matching action based on a subset of unmatched rows.

公开(公告)号：US20240069863A1

公开(公告)日：2024-02-29

申请号：US17895872

申请日：2022-08-25

Applicant: Databricks, Inc.

Inventor： Bart Samwel ,  Tathagata Das ,  Lars Kroll ,  Yijia Cui ,  Juliusz Sompolski ,  Tom Van Bussel

IPC: G06F7/14 ,  G06F16/14 ,  G06F16/16

CPC classification number: G06F7/14 ,  G06F16/148 ,  G06F16/16

Abstract: A method, system, and computer system for performing an operation with respect to a target table are disclosed. The method includes performing first, second and a third jobs, and obtaining a resulting table based at least in part on the second job resulting file(s) and third job resulting file(s). Performing the first job includes determining a set of matching target table files and storing target table information indicating for each of the set of matching target table files, a particular set of rows having matching rows. Performing the second job includes performing a matching action based on matched rows and obtaining the second job resulting file(s). Performing the third job includes determining unmatched rows for target table files and storing the unmatched rows in third job resulting file(s).

公开(公告)号：US20240061840A1

公开(公告)日：2024-02-22

申请号：US18162366

申请日：2023-01-31

Applicant: Databricks, Inc.

Inventor： Prashanth Menon ,  Alexander Behm ,  Sriram Krishnamurthy

IPC: G06F16/2453 ,  G06F16/28

CPC classification number: G06F16/24542 ,  G06F16/285

Abstract: The present application discloses a method, system, and computer system for parsing files. The method includes receiving an indication that a first file is to be processed, determining to begin processing the first file using a first processing engine based at least in part on one or more predefined heuristics, indicating to process the first file using a first processing engine, determining whether a particular error in processing the first file using the first processing engine has been detected, in response to determining that the particular error has been detected, indicate to stop processing the first file using the first processing engine and indicate to continue processing using a second processing engine, and storing in memory information obtained based on processing the first file by one or more of the first processing engine and the second processing engine.

公开(公告)号：US20240061839A1

公开(公告)日：2024-02-22

申请号：US17892376

申请日：2022-08-22

Applicant: Databricks, Inc.

Inventor： Prashanth Menon ,  Alexander Behm ,  Sriram Krishnamurthy

IPC: G06F16/2453 ,  G06F16/28

CPC classification number: G06F16/24542 ,  G06F16/285

Abstract: The present application discloses a method, system, and computer system for parsing files. The method includes receiving an indication that a first file is to be processed, determining to begin processing the first file using a first processing engine based at least in part on one or more predefined heuristics, indicating to process the first file using a first processing engine, determining whether a particular error in processing the first file using the first processing engine has been detected, in response to determining that the particular error has been detected, indicate to stop processing the first file using the first processing engine and indicate to continue processing using a second processing engine, and storing in memory information obtained based on processing the first file by one or more of the first processing engine and the second processing engine.

公开(公告)号：US20230359602A1

公开(公告)日：2023-11-09

申请号：US17738609

申请日：2022-05-06

Applicant: Databricks Inc.

Inventor： Bart Samwel ,  Prakhar Jain

IPC: G06F16/22

CPC classification number: G06F16/2246

Abstract: A system for clustering data into corresponding files comprises one or more processors and a memory. The one or more processors is/are configured to: 1) determine to cluster a set of data into a set of files; 2) determine a set of split points in a corresponding set of dimensions of the set of data to determine the set of files, wherein each file of the set of files has an approximate target size; and 3) store one or more items of the set of data into a corresponding file of the set of files based at least in part on the set of split points. The memory is coupled to the one or more processors and configured to provide the processor with instructions.

公开(公告)号：US11775499B2

公开(公告)日：2023-10-03

申请号：US17695411

申请日：2022-03-15

Applicant: Databricks, Inc.

Inventor： Michael Paul Armbrust ,  Shixiong Zhu ,  Burak Yavuz

IPC: G06F16/14 ,  G06F16/22 ,  G06F16/23

CPC classification number: G06F16/2358 ,  G06F16/148 ,  G06F16/2282

Abstract: A system includes an interface and a processor. The interface is configured to receive a table indication of a data table and to receive a transaction indication to perform a transaction. The processor is configured to determine a current position N in a transaction log; determine a current state of the metadata; determine a read set associated with a transaction; attempt to write an update to the transaction log associated with a next position N+1; in response to a transaction determination that a simultaneous transaction associated with the next position N+1 already exists, determine a set of updated files; and in response to a determination that there is not an overlap between the read set associated with the current transaction and the set of updated files associated with the simultaneous transaction, attempt to write the update to the transaction to the transaction log associated with a further position N+2.