Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CI: Cache libimagequant on Linux builds #7741

Merged
merged 4 commits into from
Jan 26, 2024
Merged

Conversation

hugovk
Copy link
Member

@hugovk hugovk commented Jan 21, 2024

On the CI, the step to "Install Linux dependencies" takes between 6m10s and 6m30s.

The step runs .ci/install.sh and its slowest commands are:

    # webp
    pushd depends && ./install_webp.sh && popd

    # libimagequant
    pushd depends && ./install_imagequant.sh && popd

    # raqm
    pushd depends && ./install_raqm.sh && popd

They take ~25s, ~4m30s and ~40s respectively.

Given the Linux jobs mostly take around 8m30s, caching these steps can save up to 5m30s, and would greatly improve the CI time.

Let's start with the slowest: libimagequant.

First, add to the workflow a cache step:

    - name: Cache libimagequant
      uses: actions/cache@v4
      id: cache-libimagequant
      with:
        path: ~/cache-libimagequant
        key: ${{ runner.os }}-libimagequant-${{ hashFiles('depends/install_imagequant.sh') }}

If there's a matching cache file, it unpacks the cache to ~/cache-libimagequant and we have a cache hit.

Then we pass in the cache hit status as an environment variable (GHA_LIBIMAGEQUANT_CACHE_HIT) to install.sh.

install.sh calls depends/install_imagequant.sh, which checks the env var.

  • If there's a cache hit, it sudo copies the cached files from ~/cache-libimagequant to /usr/....

  • If there's a cache miss, we need to build from source, as before. Except we also copy the built files into ~/cache-libimagequant so they can be cached for next time. (Delete and recreate the dir, just to be sure no old cached files are retained.)

This reduces the "Install Linux dependencies" step from 6.5-8 minutes to to 1.5-2 minutes, and saves about 45 minutes of CPU time in total.

Total time for some jobs:

 Job Before After (cached)
ubuntu-latest Python pypy3.10 13m 1s 7m 28s
ubuntu-latest Python pypy3.9 12m 29s 7m 16s
ubuntu-latest Python 3.13 9m 49s 4m 16s
ubuntu-latest Python 3.12 10m 46s 4m 21s
ubuntu-latest Python 3.11 9m 54s 4m 8s
ubuntu-latest Python 3.10 10m 24s 3m 54s
ubuntu-latest Python 3.9 8m 47s 4m 22s
ubuntu-latest Python 3.8 9m 2s 4m 4s
Total 81m 20s 37 m37s

After this, we can do a similar thing for install_raqm.sh and install_webp.sh to shave off another minute or so per job; I left them out because I wasn't sure what the filenames are that need caching; what are they?


Also:

  • Include pyproject.toml along with .ci/*.sh in the pip cache key, because we define test/docs dependencies in there too

  • Disable the wget progress bar but not all output, to omit hundreds of lines in the logs like:

     0K .......... .......... .......... .......... ..........  1% 40.7M 0s
    50K .......... .......... .......... .......... ..........  2% 38.5M 0s
   100K .......... .......... .......... .......... ..........  3% 41.2M 0s
   150K .......... .......... .......... .......... ..........  4%  218M 0s
   200K .......... .......... .......... .......... ..........  6%  160M 0s
   250K .......... .......... .......... .......... ..........  7%  130M 0s
   300K .......... .......... .......... .......... ..........  8% 75.3M 0s
   350K .......... .......... .......... .......... ..........  9%  206M 0s
   400K .......... .......... .......... .......... .......... 11%  124M 0s
   450K .......... .......... .......... .......... .......... 12%  220M 0s

@hugovk hugovk added the Testing label Jan 21, 2024
depends/install_imagequant.sh Outdated Show resolved Hide resolved
Co-authored-by: Ondrej Baranovič <nulano@nulano.eu>
@radarhere radarhere merged commit b82e601 into python-pillow:main Jan 26, 2024
55 of 57 checks passed
@hugovk hugovk deleted the ci-stuff branch January 26, 2024 16:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants