Project setup

Configuration

This project ships with an example luigi.cfg, luigi.cfg.example. You should create a copy of this file in root folder of the project that is using kiluigi, and adjust it as necessary.

Ensuring standard compliant source code files

Kimetrica uses black and flake8 to enforce correctly formatted and syntactically correct source code files. Please install the pre-commit hook into your local git clone to make sure that all files are checked whether they are standard compliant before they can be submitted to the repository.

Hook installation instructions

Just copy or link the pre-commit file at the toplevel of this repository into your .git/hooks folder. Since this repo is usually installed as a submodule, you can find out the correct path by running this command: git rev-parse --git-path hooks. Then copy the pre-commit file to this directory, e.g.:

cp -v pre-commit /PATH/TO/PARENT/REPO/.git/modules/kiluigi/hooks.git/hooks/

Testing

Testing is performed using tox. Running tox without and additional parameters will test KiLuigi against all supported Python and Pandas versions. A single Python version, with the latest supported version of Pandas, can be tested using, for example, tox -e py39.

A subset of tests can be run using a command like:

tox -e py39 tests/targets/test_ckan.py::CkanTargetTestCase::test_atomicity

Many of the tests, particularly for Target subclasses, rely on external servers. Those tests are skipped unless the appropriate envionment variable containing the server url has been set.

Testing S3 Targets

We use MinIO as a substitute for S3 in local environments. Create a copy of env.example called .env, substituting placeholder values with CKAN and Google credentials. You can now run an appropriate MinIO container using:

docker run -it --rm --env-file=.env -p 9000:9000 -p 9001:9001 --name minio minio/minio server /mnt/data --console-address ":9001"

In another shell, you can start a container for executing the tests like this:

docker run -ti --rm --env-file=.env -v ${PWD}:/usr/src/app -w /usr/src/app --link minio:minio --entrypoint=bash python:3.9

Once the container is running, you will need to create the test bucket:

apt update && apt install -y awscli
aws s3 mb ${S3_TEST_BUCKET} --endpoint-url ${S3_ENDPOINT_URL}

After the bucket has been created, run the Kiluigi tests in the Python container as follows:

pip install tox
tox -e py39 -- tests/targets/test_s3.py

The container uses Python 3.9, so you have to execute tox -e py39 (or either install additional Python versions, or use a container that contains multiple Python versions to run the full test matrix).

Testing CKAN Targets

If the CKAN_TEST_ORGANIZATION environment variable is set to the URL of an organization on a reachable CKAN server (for example https://data.kimetrica.com/test-organization), and the CKAN_TOKEN environment variable is set to a valid API token for accessing the server specified by CKAN_TEST_ORGANIZATION, then running tox will automatically include the appropriate CKAN tests.