From 97b89888b795b18ad7da00a3dfdaa3507fef08de Mon Sep 17 00:00:00 2001 From: Ishika Aahna <123102828+codedbyishika@users.noreply.github.com> Date: Mon, 13 Jul 2026 15:03:59 -0400 Subject: [PATCH 1/4] Creation of new files for design docs final-design: ideal final architecture of cautious-robot roadmap: process/tasks for the implementation of final-design --- design-doc/final-design.md | 0 design-doc/roadmap.md | 0 2 files changed, 0 insertions(+), 0 deletions(-) create mode 100644 design-doc/final-design.md create mode 100644 design-doc/roadmap.md diff --git a/design-doc/final-design.md b/design-doc/final-design.md new file mode 100644 index 0000000..e69de29 diff --git a/design-doc/roadmap.md b/design-doc/roadmap.md new file mode 100644 index 0000000..e69de29 From 058b7db1316ad3a80f8c03e9010e90781103a496 Mon Sep 17 00:00:00 2001 From: Ishika Aahna <123102828+codedbyishika@users.noreply.github.com> Date: Mon, 20 Jul 2026 16:20:06 -0400 Subject: [PATCH 2/4] Initial Document Images for the structure and control and data flow are in https://app.eraser.io/workspace/bhfPH97OkNPUhSZl0KZV --- design-doc/final-design.md | 7 +++++++ design-doc/roadmap.md | 13 +++++++++++++ 2 files changed, 20 insertions(+) diff --git a/design-doc/final-design.md b/design-doc/final-design.md index e69de29..e5b0b85 100644 --- a/design-doc/final-design.md +++ b/design-doc/final-design.md @@ -0,0 +1,7 @@ +## New Design Implementations + +- Parallel Downloading: +Parallel downloading faster and efficient downloading of images from user csv input rather than a serial. Using methods of multiple HTTP connections and Additive Increase and Multiplicative Decrease. + +- Data Output in Different Formats: +User can select the final output of the files to be saved such as archived zip files or web dataset. This will allow the user to have more versatality of the final output for the images downloaded. diff --git a/design-doc/roadmap.md b/design-doc/roadmap.md index e69de29..5e2bdf3 100644 --- a/design-doc/roadmap.md +++ b/design-doc/roadmap.md @@ -0,0 +1,13 @@ +> **Validation module** `**rollcall**` +Checking for pre-download requirements to ensure that the download takes place without any interruptions or failure because of the following conditions: +- Invalid csv extension +- Duplicate filenames +- Missing filenames +- Existing images on drive +- Duplicate checksums + +> **Image Processing module** `**download**` +Downloading of images according csv provided by user and initial rollcall checks. + +> **Verification module** `**buddy_check**` +Post-download verification of the downloaded images including checks for checksums and filenames. From 6ae91ade11b64cde2788473d96f091b33f1bbea0 Mon Sep 17 00:00:00 2001 From: Ishika Aahna <123102828+codedbyishika@users.noreply.github.com> Date: Tue, 21 Jul 2026 14:03:26 -0400 Subject: [PATCH 3/4] Include more description on functions --- design-doc/roadmap.md | 31 ++++++++++++++++++++++++++++--- 1 file changed, 28 insertions(+), 3 deletions(-) diff --git a/design-doc/roadmap.md b/design-doc/roadmap.md index 5e2bdf3..03355f7 100644 --- a/design-doc/roadmap.md +++ b/design-doc/roadmap.md @@ -1,4 +1,7 @@ -> **Validation module** `**rollcall**` +# __Current Design__ + +### Modularization +> **Validation module** `rollcall` Checking for pre-download requirements to ensure that the download takes place without any interruptions or failure because of the following conditions: - Invalid csv extension - Duplicate filenames @@ -6,8 +9,30 @@ Checking for pre-download requirements to ensure that the download takes place w - Existing images on drive - Duplicate checksums -> **Image Processing module** `**download**` +> **Image Processing module** `download` Downloading of images according csv provided by user and initial rollcall checks. -> **Verification module** `**buddy_check**` +> **Verification module** `buddy_check` Post-download verification of the downloaded images including checks for checksums and filenames. + +### Functions & Contracts +`rollcall module` +- __ init __ +- validate_csv_extension +- validate_filename_uniqueness +- handle_missing_filenames +- setup_expected_columns +- check_existing_images +- _preview_or_save +- check_duplicate_checksums +- print_download_summary + +`download module` +- downsample_and_save + +`buddy_check module` +- __ init __ +- merge_on_checksum +- merge_on_filename_checksum +- process_checksums +- verify_downloads \ No newline at end of file From 39162ac09fbe9886896a38a4715df2739e2d6b19 Mon Sep 17 00:00:00 2001 From: Ishika Aahna <123102828+codedbyishika@users.noreply.github.com> Date: Tue, 21 Jul 2026 14:12:08 -0400 Subject: [PATCH 4/4] Architecture Update --- design-doc/roadmap.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/design-doc/roadmap.md b/design-doc/roadmap.md index 03355f7..704496c 100644 --- a/design-doc/roadmap.md +++ b/design-doc/roadmap.md @@ -35,4 +35,7 @@ Post-download verification of the downloaded images including checks for checksu - merge_on_checksum - merge_on_filename_checksum - process_checksums -- verify_downloads \ No newline at end of file +- verify_downloads + +### Architecture +The struture of the files and functions after modularization and the control and data flows can be accessed [here](https://app.eraser.io/workspace/bhfPH97OkNPUhSZl0KZV). \ No newline at end of file