Skip to content

feat(outputs): add dedicated Snowpipe Streaming output plugin for Snowflake #18735

@aleksclark

Description

@aleksclark

Use Case

Ingesting Telegraf metrics into Snowflake for analytics, dashboarding, and long-term storage. The existing SQL output plugin can connect to Snowflake via gosnowflake, but it is a generic SQL driver with no awareness of Snowflake-specific optimizations: no array binding for batch inserts, no automatic table creation or schema evolution, no Snowpipe Streaming integration, and no template-based table routing.

A dedicated Snowpipe Streaming plugin provides production-grade Snowflake ingestion with efficient batch inserts (array binding), key-pair authentication, automatic DDL, template-based table routing (e.g. metrics_{{.Name}}), retry with exponential backoff, and NaN/Inf sanitization.

Expected behavior

A dedicated outputs.snowpipe_streaming plugin that:

  • Authenticates via RSA key-pair (Snowflake best practice for service accounts)
  • Routes metrics to tables using Go templates based on metric name
  • Batches inserts efficiently using gosnowflake array binding
  • Automatically creates tables and evolves schema as new fields appear
  • Retries transient errors with exponential backoff
  • Sanitizes NaN/Inf values to NULL
  • Caches schema with configurable TTL to minimize DESCRIBE TABLE calls

Actual behavior

No dedicated Snowflake output plugin exists. Users must use the generic SQL output plugin which lacks batch optimization, schema management, table routing, and Snowpipe Streaming awareness. This results in suboptimal insert performance and requires manual table creation and schema management.

Additional info

A PR implementing this plugin is available: #18721

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions