packages/sources owns reusable dlt source definitions. It is where external
systems enter the analytics workspace.
Use this package when adding or changing how Pyronear Analytics reads from an
external system. Keep workflow-specific decisions in packages such as pyromap;
sources should describe how to extract source records, not how to transform or
publish them.
dlt is the Python library this repository uses for ingestion. In dlt terms, a source groups one or more resources, and a resource describes records that can be loaded into a destination table.
dlt handles common ingestion concerns such as REST API extraction, authentication, normalization, schema inference, loading, and access to the loaded dataset.
In this repository, dlt stops at ingestion. A dlt pipeline loads private source records into a configured destination such as DuckDB. Workflow packages read those loaded rows and perform validation, transformation, privacy checks, serialization, and publication outside dlt.
That split keeps source definitions reusable. The same backend source can be used by Pyromap today and by future workflows later without importing map privacy rules, GeoJSON serialization, or publisher settings.
The package is responsible for:
- defining dlt sources and resources
- describing backend API access
- keeping credentials and runtime destination choices outside package code
The package must not contain:
- workflow-specific transformation logic
- privacy or public-artifact policy
- GeoJSON serialization
- local or S3 publication logic
The currently implemented source is sources.backend.backend_source, which
exposes the Pyronear backend cameras resource.
For the implemented source contract, see Pyronear API.