diff --git a/design-doc/final-design.md b/design-doc/final-design.md new file mode 100644 index 0000000..fad0c9e --- /dev/null +++ b/design-doc/final-design.md @@ -0,0 +1,7 @@ +## New Design Implementations + +- Parallel Downloading: +Parallel downloading faster and efficient downloading of images from user csv input rather than a serial. Using methods of multiple HTTP connections and Additive Increase and Multiplicative Decrease. + +- Data Output in Different Formats (Potential Option if Use Case): +User can select the final output of the files to be saved such as archived zip files or Parquet. This will allow the user to have more versatility of the final output for the images downloaded. diff --git a/design-doc/roadmap.md b/design-doc/roadmap.md new file mode 100644 index 0000000..8c1e85c --- /dev/null +++ b/design-doc/roadmap.md @@ -0,0 +1,49 @@ +# __Current Design__ + +### Modularization +> **Validation module** `rollcall` +Checking for pre-download requirements to ensure that the download takes place without any interruptions or failure because of the following conditions: +- Invalid csv extension +- Duplicate filenames +- Missing filenames +- Existing images on drive +- Duplicate checksums + +> **Image Processing module** `download` +Downloading of images according csv provided by user and initial rollcall checks. + +> **Verification module** `buddy_check` +Post-download verification of the downloaded images including checks for checksums and filenames. + +### Functions & Contracts +`rollcall module` +- __ init __ +- validate_csv_extension +- validate_filename_uniqueness +- handle_missing_filenames +- setup_expected_columns +- check_existing_images +- _preview_or_save +- check_duplicate_checksums +- print_download_summary + +`download module` +- downsample_and_save + +`buddy_check module` +- __ init __ +- merge_on_checksum +- merge_on_filename_checksum +- process_checksums +- verify_downloads + +### Architecture +The struture of the files and functions after modularization and the control and data flows can be accessed [here](https://app.eraser.io/workspace/bhfPH97OkNPUhSZl0KZV). + +> Include flag to allow uset to sequential download instead of parellel downloads. + + +### Agents for Parellel Downloading +- Scheduler: Inputs from rollcall and processes the downloads based on the rate limit of servers and http headers. +- Orchestrator/Watcher: Determines number of workers for download, monitors time stamps and rate of download and additively increases or decreases cautiously based on the rate limit. +- Workers: Downloads images with set rate limit. \ No newline at end of file