Skip to content

feat: added automated ingestion workflow #113

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
vishpillai123 merged 45 commits into from
Mar 2, 2026
Merged

feat: added automated ingestion workflow #113

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
a6f5c9c
Merge branch 'release/0.1.7'
Dec 3, 2025
3d74559
Merge branch 'release/0.1.8'
Dec 11, 2025
187d54e
Merge branch 'release/0.1.9'
Jan 20, 2026
b2d8c1a
Merge branch 'release/0.1.10'
Feb 2, 2026
0b24b2c
feat: added automated ingestion workflow
Feb 19, 2026
ea9ea9d
Merge branch 'develop' into Automated_Ingestion_Workflow
Mesh-ach Feb 19, 2026
7b5ce17
added pdp ingestion files
Mesh-ach Feb 19, 2026
34297b2
Merge branch 'develop' into Automated_Ingestion_Workflow
Mesh-ach Feb 23, 2026
649ef40
feat: moved reusueable components into helper.py
Mesh-ach Feb 23, 2026
716ac97
fix: initialized spark
Mesh-ach Feb 23, 2026
d3e0f74
fix: initialized spark
Mesh-ach Feb 24, 2026
0a3ae3a
fix: notebook docs
Mesh-ach Feb 24, 2026
42357b7
fix: notebook docs
Mesh-ach Feb 24, 2026
9b137e9
feat: refactor
Feb 24, 2026
ca4ef23
refactor: moved helpers to src code
Feb 24, 2026
8ef2dea
refactor: putting hardcoded constants in constants file
Feb 24, 2026
318c65b
refactor: moved functions from notebook 1 into modules
Feb 24, 2026
953d350
Add comprehensive test class for databricksify_inst_name
Feb 24, 2026
87089b1
fix: import
Feb 24, 2026
fd557e9
fix: tests & style
Feb 24, 2026
f6197da
fix: ruff
Feb 24, 2026
180e4e9
fix: style
Feb 24, 2026
0b2d742
fix: type check with overrides for paramiko
Feb 24, 2026
4c061a7
fix: type check
Feb 24, 2026
e64280c
chore: move test file from notebooks/ to ingestion/
Feb 24, 2026
1d30428
Move databricksify_inst_name and reverse_databricksify_inst_name to u…
Feb 24, 2026
a97fdbb
Move databricksify tests to test_databricks.py
Feb 24, 2026
c83c2bd
fix: style
Feb 24, 2026
d3fee8e
style
Feb 24, 2026
2973bab
fix: tests
Feb 24, 2026
54c979b
fix: added env differentiation
Mesh-ach Feb 26, 2026
dc68cf9
fix: env path
Mesh-ach Feb 26, 2026
1e81e88
fix: mandatory databricks parameters
Mesh-ach Feb 26, 2026
7af1c5b
fix: claude review
Mesh-ach Feb 26, 2026
92c78ba
fix: claude review
Mesh-ach Feb 26, 2026
c929ff1
fix: claude review
Mesh-ach Feb 26, 2026
616cee4
fix: added edvise imports
Mesh-ach Feb 26, 2026
e35ceba
fix: added verify parameter to download_sftp_atomic function
Mesh-ach Feb 26, 2026
d5c1e6e
fix: issues with snakecase normalizations that claude flagged
Mesh-ach Feb 26, 2026
12c5287
fix: resolved dbutils issues
Mesh-ach Feb 26, 2026
e17d074
fix: resolved dbutils issues
Mesh-ach Feb 26, 2026
271de56
fix: resolved gcp_config.ysml
Mesh-ach Feb 26, 2026
bd8f442
fix: resolved ruff issues
Mesh-ach Feb 26, 2026
44c77bf
fix: resolved ruff issues
Mesh-ach Feb 26, 2026
25af159
fix: added valuable output statements for workflow
Mesh-ach Feb 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -212,3 +212,4 @@ __marimo__/

# Claude
.claude/
*notebooks/nsc_sftp_automated_data_ingestion/tmp/
Loading
Loading