Dynamic DICOM Endpoints Project discussion

[atulbhardwaj-io]

Title: Proposal: Diomede Album Management Architecture

Hi @pradeeban and mentors,

After reading the described idea and past discussions, I would like to propose an album-focused implementation that follows the clarified scope of Diomede.

My understanding is that Diomede should focus on creating and managing albums for locally available DICOM images, while reusing existing metadata extraction and query workflows (such as those in Niffler) instead of rebuilding them.

Proposed Architecture

Metadata Query Layer
Query stored DICOM metadata (Modality, PatientID, StudyInstanceUID, SeriesInstanceUID, SOPInstanceUID, date ranges, DICOM source) with filtering, pagination, and the ability to select multiple entries to add to an album.
The system will follow the DICOM hierarchy:

PatientID → StudyInstanceUID → SeriesInstanceUID → SOPInstanceUID

No additional grouping structures will be introduced.

Album Management
Create, rename, and delete albums.
Add or remove DICOM instances.
Create albums directly from query results.
Store Kheops share URLs and synchronization status.

Storage Design
Use a relational database for the initial version.
Optionally support a versioned metadata backend (e.g., Delta Lake) to handle incremental ingestion, repeated scans without duplication, audit tracking, and reproducible query results for larger datasets.
DICOM files will remain stored on the filesystem.

Kheops Integration
Create albums in Kheops, upload selected DICOM files, track upload status, and handle retry logic separately from the main album logic.

Since I work in data engineering, I can also contribute to metadata ingestion design, incremental processing, schema design for efficient querying, and version tracking.

Delivery Plan:

Phase 1 – Metadata query and local album management
Phase 2 – Kheops integration
Phase 3 – Source management and incremental rescans
Phase 4 – Testing and documentation

If this direction matches your expectations, I can share a more detailed technical design and timeline.

Thank you.

[pradeeban]

sounds good, @atulbhardwaj-io!