REAL-TIME STREAMING DATA INGESTION INTO DATABASE TABLES

    公开(公告)号:US20220327132A1

    公开(公告)日:2022-10-13

    申请号:US17647500

    申请日:2022-01-10

    Applicant: Snowflake Inc.

    Abstract: A streaming ingest platform can improve latency and expense issues related to uploading data into a cloud data system. The streaming ingest platform can organize the data to be ingested into per-table chunks and per-account blobs. This data may be committed and may be made available for query processing before it is ingested into the target source tables. This significantly improves latency issues. The streaming ingest platform can also accommodate uploading data from various sources with different processing and communication capabilities, such as Internet of Things (IOT) devices.

    Real-time streaming data ingestion into database tables

    公开(公告)号:US11250006B1

    公开(公告)日:2022-02-15

    申请号:US17386258

    申请日:2021-07-27

    Applicant: Snowflake Inc.

    Abstract: A streaming ingest platform can improve latency and expense issues related to uploading data into a cloud data system. The streaming ingest platform can organize the data to be ingested into per-table chunks and per-account blobs. This data may be committed and may be made available for query processing before it is ingested into the target source tables. This significantly improves latency issues. The streaming ingest platform can also accommodate uploading data from various sources with different processing and communication capabilities, such as Internet of Things (IOT) devices.

    Real-time streaming data ingestion into database tables

    公开(公告)号:US11100111B1

    公开(公告)日:2021-08-24

    申请号:US17226423

    申请日:2021-04-09

    Applicant: Snowflake Inc.

    Abstract: A streaming ingest platform can improve latency and expense issues related to uploading data into a cloud data system. The streaming ingest platform can organize the data to be ingested into per-table chunks and per-account blobs. This data may be committed and may be made available for query processing before it is ingested into the target source tables. This significantly improves latency issues. The streaming ingest platform can also accommodate uploading data from various sources with different processing and communication capabilities, such as Internet of Things (IOT) devices.

    QUERY EXECUTION USING INTERMEDIATE MATERIALIZED TABLES

    公开(公告)号:US20250086168A1

    公开(公告)日:2025-03-13

    申请号:US18955789

    申请日:2024-11-21

    Applicant: Snowflake Inc.

    Abstract: A method for query execution includes configuring, by at least one hardware processor, a materialized table (MT) based on the query and a base table. The MT includes a lag duration indicating a maximum time period that a result of a prior refresh of the query on the base table can lag behind a current time instance. The method further includes generating a query plan for the query. The method further includes determining multiple sets of data processing operations included in the query plan. The method further includes generating a plurality of intermediate MTs. The one or more intermediate MTs hold an intermediate processing state for the multiple sets of data processing operations. The method further includes configuring a refresh of the MT based on the intermediate processing state for the multiple sets of data processing operations.

    Materialized table refresh using multiple processing pipelines

    公开(公告)号:US12216654B2

    公开(公告)日:2025-02-04

    申请号:US18362898

    申请日:2023-07-31

    Applicant: Snowflake Inc.

    Abstract: A system for a materialized table (MT) refresh using multiple processing pipelines includes at least one hardware processor coupled to memory storing instructions. The instructions cause the at least one hardware processor to perform operations including determining dependencies among a plurality of intermediate MTs generated from a source MT. The source MT uses a table definition with a query on one or more base tables and a lag duration value. A graph snapshot of dependencies among the plurality of intermediate MTs is generated. Processing pipelines are configured. Each of the processing pipelines corresponds to a subset of the plurality of intermediate MTs indicated by the graph snapshot. Responsive to detecting an instruction for a refresh operation on the source MT, refreshes on corresponding intermediate MTs of the plurality of intermediate MTs in each processing pipeline of the processing pipelines are performed to complete the refresh operation on the source MT.

Patent Agency Ranking