rdbms2s3 is an ETL tool which imports data from related database to AWS S3 bucket as CSV/parquet. Currently it can handle oracle and MySQl database.
rdbms2s3 contains a few CLI tools for each different database types. User can use a config yaml file to specify database and s3 informations. Two tools are currently created for Oracle and MySQL databases.
- oracle_2_s3
- mysql_2_s3
- Create a virtual python environment. Ex: With Conda
conda create -n [name-of-environment]
conda activate [name-of-environment]
conda install pip
- cd into where the setup.py of rdbms2s3 is at and inall
cd /to/where/this/is/at/rdbms2s3
pip install .
rdbms2s3 currently only support yaml config file.
- rdbms2s3.yaml.
# database info
database:
type: oracle # oracle/mysql
url: ****************
schema: *******
password: ********
tables:
- table1
- table2
- view1
# s3 setting
s3:
region_name: *****
bucket_name: ***s3-bucket**
folder_name: ******
access_key_id: ******
secret_access_key: *************************
# only use for oracle database
oracle:
instant_client: /path/to/instantclient
- Install the oracle instant client (Save the location in config file)
- Create config file. (Use
rdbms2s3.yamlfrom resources for reference). - To run the application:
oracle_2_s3 -p /path/to/config/file
- Create config file. (Use
rdbms2s3.yamlfrom resources for reference). - To run the application:
mysql_2_s3 -p /path/to/config/file