Skip to main content

AzureQueue Table Engine

This engine provides an integration with Azure Blob Storage ecosystem, allowing streaming data import.

Create Table​

CREATE TABLE test (name String, value UInt32)
    ENGINE = AzureQueue(...)
    [SETTINGS]
    [mode = '',]
    [after_processing = 'keep',]
    [keeper_path = '',]
    ...

Engine parameters

AzureQueue parameters are the same as AzureBlobStorage table engine supports. See parameters section here.

Example

CREATE TABLE azure_queue_engine_table (name String, value UInt32)
ENGINE=AzureQueue('DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://azurite1:10000/devstoreaccount1/data/')
SETTINGS
    mode = 'unordered'

Settings​

The set of supported settings is the same as for S3Queue table engine, but without s3queue_ prefix. See full list of settings settings. To get a list of settings, configured for the table, use system.s3_queue_settings table. Available from 24.10.

Description​

SELECT is not particularly useful for streaming import (except for debugging), because each file can be imported only once. It is more practical to create real-time threads using materialized views. To do this:

  1. Use the engine to create a table for consuming from specified path in S3 and consider it a data stream.
  2. Create a table with the desired structure.
  3. Create a materialized view that converts data from the engine and puts it into a previously created table.

When the MATERIALIZED VIEW joins the engine, it starts collecting data in the background.

Example:

  CREATE TABLE azure_queue_engine_table (name String, value UInt32)
    ENGINE=AzureQueue('<endpoint>', 'CSV', 'gzip')
    SETTINGS
        mode = 'unordered';

  CREATE TABLE stats (name String, value UInt32)
    ENGINE = MergeTree() ORDER BY name;

  CREATE MATERIALIZED VIEW consumer TO stats
    AS SELECT name, value FROM azure_queue_engine_table;

  SELECT * FROM stats ORDER BY name;

Virtual columns​

  • _path — Path to the file.
  • _file — Name of the file.

For more information about virtual columns see here.