Invention Grant
- Patent Title: System and method for handling data skew at run time
-
Application No.: US16164756Application Date: 2018-10-18
-
Publication No.: US10713096B2Publication Date: 2020-07-14
- Inventor: Shouwei Chen , Wensheng Wang
- Applicant: Beijing Jingdong Shangke Information Technology Co., Ltd. , JD.com American Technologies Corporation
- Applicant Address: CN Beijing US CA Mountain View
- Assignee: Beijing Jingdong Shangke Information Technology Co., Ltd.,JD.com American Technologies Corporation
- Current Assignee: Beijing Jingdong Shangke Information Technology Co., Ltd.,JD.com American Technologies Corporation
- Current Assignee Address: CN Beijing US CA Mountain View
- Agency: China Science Patent & Trademark US LLC
- Main IPC: G06F15/16
- IPC: G06F15/16 ; G06F9/50 ; G06F3/06 ; G06F9/38

Abstract:
A system for handling data skew includes a cluster of computing nodes and a processor. The cluster includes one or more first nodes, each of which has a storage, and one or more second nodes, each of which has a storage. The storage of the respective second nodes has a higher access speed than the storage of the respective first nodes. The processor is configured to split input data into partitions of the input data, to detect if any of the partitions has data skew, and to assign ones of the partitions, which are detected as having no data skew, to the first nodes, and ones of the partitions, which are detected as having the data skew, to the second nodes, for parallel processing.
Public/Granted literature
- US20200125417A1 SYSTEM AND METHOD FOR HANDLING DATA SKEW AT RUN TIME Public/Granted day:2020-04-23
Information query