Skip to content

Prefetch #1397

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
shinnar wants to merge 2 commits into from
Closed

Prefetch #1397

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
285123e
Add prefetching of sklearn data. Change cache directory so that is i…
shinnar Apr 2, 2025
0635206
Change github actions to use setup-python cache instead of separate one
shinnar Apr 2, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
145 changes: 41 additions & 104 deletions .github/workflows/build.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,8 @@ name: Tests

on: [push, pull_request, workflow_dispatch]
env:
LALE_DOWNLOAD_CACHE_DIR: ${{ github.workspace }}/.cache/data
DOWNLOAD_CACHE_DIR: /home/runner/scikit_learn_data
LALE_DOWNLOAD_CACHE_DIR: /home/runner/scikit_learn_data/lale

jobs:
static:
Expand All @@ -19,17 +20,9 @@ jobs:
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
with:
cache: 'pip'
cache-dependency-path: setup.py
python-version: ${{ matrix.python-version }}
- name: Cache pip
uses: actions/cache@v4
with:
# This path is specific to Ubuntu
path: ${{ env.pythonLocation }}
# Look to see if there is a cache hit for the setup file
key: ${{ runner.os }}-pip-new3-${{ env.pythonLocation }}-${{ hashFiles('setup.py') }}
restore-keys: |
${{ runner.os }}-pip-new3
${{ runner.os }}-new3
- name: Install numpy
run: pip install -U numpy
- name: Install dependencies
Expand All @@ -53,16 +46,8 @@ jobs:
uses: actions/setup-python@v5
with:
python-version: 3.9
- name: Cache pip
uses: actions/cache@v4
with:
# This path is specific to Ubuntu
path: ${{ env.pythonLocation }}
# Look to see if there is a cache hit for the setup file
key: ${{ runner.os }}-pip-new3-${{ env.pythonLocation }}-${{ hashFiles('setup.py') }}
restore-keys: |
${{ runner.os }}-pip-new3
${{ runner.os }}-new3
cache: 'pip'
cache-dependency-path: setup.py
- name: Install doc dependencies
run: pip install -r docs/requirements.txt
- name: Install lale
Expand All @@ -89,16 +74,8 @@ jobs:
uses: actions/setup-python@v5
with:
python-version: '3.11'
- name: Cache pip
uses: actions/cache@v4
with:
# This path is specific to Ubuntu
path: ${{ env.pythonLocation }}
# Look to see if there is a cache hit for the setup file
key: ${{ runner.os }}-pip-new3-${{ env.pythonLocation }}-${{ hashFiles('setup.py') }}
restore-keys: |
${{ runner.os }}-pip-new3
${{ runner.os }}-new3
cache: 'pip'
cache-dependency-path: setup.py
- name: Install numpy
run: pip install -U numpy
- name: Install dependencies
Expand All @@ -112,18 +89,18 @@ jobs:
- name: Cache test data restore
uses: actions/cache/restore@v4
with:
path: ${{ env.LALE_DOWNLOAD_CACHE_DIR }}
key: ${{ runner.os }}-dcache-new3-${{ env.LALE_DOWNLOAD_CACHE_DIR }}-${{ hashFiles('lale/datasets/prefetch.py') }}
path: ${{ env.DOWNLOAD_CACHE_DIR }}
key: ${{ runner.os }}-dcache-new3-${{ env.DOWNLOAD_CACHE_DIR }}-${{ hashFiles('lale/datasets/prefetch.py') }}
restore-keys: |
${{ runner.os }}-dcache-new3-${{ env.LALE_DOWNLOAD_CACHE_DIR }}-
${{ runner.os }}-dcache-new3-${{ env.DOWNLOAD_CACHE_DIR }}-
${{ runner.os }}-dcache-new3
- name: Download the test data
run: python lale/datasets/prefetch.py
- name: Cache test data save
uses: actions/cache/save@v4
with:
path: ${{ env.LALE_DOWNLOAD_CACHE_DIR }}
key: ${{ runner.os }}-dcache-new3-${{ env.LALE_DOWNLOAD_CACHE_DIR }}-${{ hashFiles('lale/datasets/prefetch.py') }}
path: ${{ env.DOWNLOAD_CACHE_DIR }}
key: ${{ runner.os }}-dcache-new3-${{ env.DOWNLOAD_CACHE_DIR }}-${{ hashFiles('lale/datasets/prefetch.py') }}


# test_matrix_pr and test_matrix_master have a lot of redundancy
Expand Down Expand Up @@ -187,25 +164,17 @@ jobs:
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
cache: 'pip'
cache-dependency-path: setup.py
- name: Install system packages
run: sudo apt-get install graphviz swig
- name: Cache pip
uses: actions/cache@v4
with:
# This path is specific to Ubuntu
path: ${{ env.pythonLocation }}
# Look to see if there is a cache hit for the setup file
key: ${{ runner.os }}-pip-new3-${{ env.pythonLocation }}-${{ hashFiles('setup.py') }}
restore-keys: |
${{ runner.os }}-pip-new3
${{ runner.os }}-new3
- name: Cache test data restore
uses: actions/cache/restore@v4
with:
path: ${{ env.LALE_DOWNLOAD_CACHE_DIR }}
key: ${{ runner.os }}-dcache-new3-${{ env.LALE_DOWNLOAD_CACHE_DIR }}-${{ hashFiles('lale/datasets/prefetch.py') }}
path: ${{ env.DOWNLOAD_CACHE_DIR }}
key: ${{ runner.os }}-dcache-new3-${{ env.DOWNLOAD_CACHE_DIR }}-${{ hashFiles('lale/datasets/prefetch.py') }}
restore-keys: |
${{ runner.os }}-dcache-new3-${{ env.LALE_DOWNLOAD_CACHE_DIR }}-
${{ runner.os }}-dcache-new3-${{ env.DOWNLOAD_CACHE_DIR }}-
${{ runner.os }}-dcache-new3
- name: Install numpy
run: pip install -U numpy
Expand Down Expand Up @@ -285,25 +254,17 @@ jobs:
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
cache: 'pip'
cache-dependency-path: setup.py
- name: Install system packages
run: sudo apt-get install graphviz swig
- name: Cache pip
uses: actions/cache@v4
with:
# This path is specific to Ubuntu
path: ${{ env.pythonLocation }}
# Look to see if there is a cache hit for the setup file
key: ${{ runner.os }}-pip-new3-${{ env.pythonLocation }}-${{ hashFiles('setup.py') }}
restore-keys: |
${{ runner.os }}-pip-new3
${{ runner.os }}-new3
- name: Cache test data restore
uses: actions/cache/restore@v4
with:
path: ${{ env.LALE_DOWNLOAD_CACHE_DIR }}
key: ${{ runner.os }}-dcache-new3-${{ env.LALE_DOWNLOAD_CACHE_DIR }}-${{ hashFiles('lale/datasets/prefetch.py') }}
path: ${{ env.DOWNLOAD_CACHE_DIR }}
key: ${{ runner.os }}-dcache-new3-${{ env.DOWNLOAD_CACHE_DIR }}-${{ hashFiles('lale/datasets/prefetch.py') }}
restore-keys: |
${{ runner.os }}-dcache-new3-${{ env.LALE_DOWNLOAD_CACHE_DIR }}-
${{ runner.os }}-dcache-new3-${{ env.DOWNLOAD_CACHE_DIR }}-
${{ runner.os }}-dcache-new3
- name: Install numpy
run: pip install -U numpy
Expand Down Expand Up @@ -343,25 +304,17 @@ jobs:
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
cache: 'pip'
cache-dependency-path: setup.py
- name: Install system packages
run: sudo apt-get install graphviz swig
- name: Cache pip
uses: actions/cache@v4
with:
# This path is specific to Ubuntu
path: ${{ env.pythonLocation }}
# Look to see if there is a cache hit for the setup file
key: ${{ runner.os }}-pip-new3-${{ env.pythonLocation }}-${{ hashFiles('setup.py') }}
restore-keys: |
${{ runner.os }}-pip-new3
${{ runner.os }}-new3
- name: Cache test data restore
uses: actions/cache/restore@v4
with:
path: ${{ env.LALE_DOWNLOAD_CACHE_DIR }}
key: ${{ runner.os }}-dcache-new3-${{ env.LALE_DOWNLOAD_CACHE_DIR }}-${{ hashFiles('lale/datasets/prefetch.py') }}
path: ${{ env.DOWNLOAD_CACHE_DIR }}
key: ${{ runner.os }}-dcache-new3-${{ env.DOWNLOAD_CACHE_DIR }}-${{ hashFiles('lale/datasets/prefetch.py') }}
restore-keys: |
${{ runner.os }}-dcache-new3-${{ env.LALE_DOWNLOAD_CACHE_DIR }}-
${{ runner.os }}-dcache-new3-${{ env.DOWNLOAD_CACHE_DIR }}-
${{ runner.os }}-dcache-new3
- name: Install numpy
run: pip install -U numpy
Expand Down Expand Up @@ -413,23 +366,15 @@ jobs:
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
- name: Cache pip
uses: actions/cache@v4
with:
# This path is specific to Ubuntu
path: ${{ env.pythonLocation }}
# Look to see if there is a cache hit for the setup file
key: ${{ runner.os }}-pip-new3-${{ env.pythonLocation }}-${{ hashFiles('setup.py') }}
restore-keys: |
${{ runner.os }}-pip-new3
${{ runner.os }}-new3
cache: 'pip'
cache-dependency-path: setup.py
- name: Cache test data restore
uses: actions/cache/restore@v4
with:
path: ${{ env.LALE_DOWNLOAD_CACHE_DIR }}
key: ${{ runner.os }}-dcache-new3-${{ env.LALE_DOWNLOAD_CACHE_DIR }}-${{ hashFiles('lale/datasets/prefetch.py') }}
path: ${{ env.DOWNLOAD_CACHE_DIR }}
key: ${{ runner.os }}-dcache-new3-${{ env.DOWNLOAD_CACHE_DIR }}-${{ hashFiles('lale/datasets/prefetch.py') }}
restore-keys: |
${{ runner.os }}-dcache-new3-${{ env.LALE_DOWNLOAD_CACHE_DIR }}-
${{ runner.os }}-dcache-new3-${{ env.DOWNLOAD_CACHE_DIR }}-
${{ runner.os }}-dcache-new3
- name: Install packages
run: sudo apt-get install graphviz swig
Expand Down Expand Up @@ -506,23 +451,15 @@ jobs:
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
- name: Cache pip
uses: actions/cache@v4
with:
# This path is specific to Ubuntu
path: ${{ env.pythonLocation }}
# Look to see if there is a cache hit for the setup file
key: ${{ runner.os }}-pip-new3-${{ env.pythonLocation }}-${{ hashFiles('setup.py') }}
restore-keys: |
${{ runner.os }}-pip-new3
${{ runner.os }}-new3
cache: 'pip'
cache-dependency-path: setup.py
- name: Cache test data restore
uses: actions/cache/restore@v4
with:
path: ${{ env.LALE_DOWNLOAD_CACHE_DIR }}
key: ${{ runner.os }}-dcache-new3-${{ env.LALE_DOWNLOAD_CACHE_DIR }}-${{ hashFiles('lale/datasets/prefetch.py') }}
path: ${{ env.DOWNLOAD_CACHE_DIR }}
key: ${{ runner.os }}-dcache-new3-${{ env.DOWNLOAD_CACHE_DIR }}-${{ hashFiles('lale/datasets/prefetch.py') }}
restore-keys: |
${{ runner.os }}-dcache-new3-${{ env.LALE_DOWNLOAD_CACHE_DIR }}-
${{ runner.os }}-dcache-new3-${{ env.DOWNLOAD_CACHE_DIR }}-
${{ runner.os }}-dcache-new3
- name: Install packages
run: sudo apt-get install graphviz swig
Expand Down Expand Up @@ -580,10 +517,10 @@ jobs:
- name: Cache test data restore
uses: actions/cache/restore@v4
with:
path: ${{ env.LALE_DOWNLOAD_CACHE_DIR }}
key: ${{ runner.os }}-dcache-new3-${{ env.LALE_DOWNLOAD_CACHE_DIR }}-${{ hashFiles('lale/datasets/prefetch.py') }}
path: ${{ env.DOWNLOAD_CACHE_DIR }}
key: ${{ runner.os }}-dcache-new3-${{ env.DOWNLOAD_CACHE_DIR }}-${{ hashFiles('lale/datasets/prefetch.py') }}
restore-keys: |
${{ runner.os }}-dcache-new3-${{ env.LALE_DOWNLOAD_CACHE_DIR }}-
${{ runner.os }}-dcache-new3-${{ env.DOWNLOAD_CACHE_DIR }}-
${{ runner.os }}-dcache-new3
- name: Run test
run: py.test -v --capture=tee-sys ${{matrix.test-case}}
Expand Down
12 changes: 12 additions & 0 deletions lale/datasets/prefetch.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@
# See the License for the specific language governing permissions and
# limitations under the License.

import sklearn.datasets

import lale.lib.aif360.datasets
from lale.datasets.movie_review import load_movie_review
from lale.datasets.multitable.fetch_datasets import fetch_go_sales_dataset
Expand Down Expand Up @@ -93,6 +95,16 @@ def prefetch_data():

fetch_fairness_dbs()

sklearn.datasets.fetch_california_housing()
sklearn.datasets.load_digits()
sklearn.datasets.load_iris()
sklearn.datasets.fetch_20newsgroups()
sklearn.datasets.load_diabetes()
sklearn.datasets.fetch_covtype()
sklearn.datasets.load_diabetes()
sklearn.datasets.fetch_openml(name="house_prices", as_frame=True)
sklearn.datasets.load_breast_cancer()


def main():
prefetch_data()
Expand Down
Loading