Caching Homebrew bottles and formulas on Travis CI

Question

I need to install a newer version of a dependency via Brew than what is contained in the Travis CI image. Running brew update adds more than 100 seconds to every build job, so I'd like to find a way to cache the installation, or at least cache the Bottle and Formula.

It seems like there should be a simple recipe for caching the Bottles and Formulas from Brew, but so far I have not found that recipe.

To be specific, I'm using pyenv on TravisCI and for the build jobs that runs tests with Python 2.6 I need pyenv to be the latest (1.2.4), since get_pip fails on earlier versions due to recent changes on PyPI.

I added a conditional to my .travis.yml and it works fine:

if [[ "$pyver" == 2.6 ]]; then
  brew upgrade pyenv &>/dev/null
fi

However, the command takes a while to run. I've split out the upgrade command and timed it:

time brew update
time brew upgrade pyenv

The brew update takes more than 100 seconds, the brew upgrade takes 10 seconds.

That finding led to the idea to cache the Bottle and Formula and on every subsequent build run:

HOMEBREW_NO_AUTO_UPDATE=1 brew upgrade pyenv

There's a question on how to cache Brew sources. That works fine.

cache:
  directories:
    - "$HOME/Library/Caches/Homebrew"

In $HOME/Library/Caches/Homebrew, I find the bottle cached:

pyenv-1.1.5.sierra.bottle.tar.gz
pyenv-1.2.4.sierra.bottle.tar.gz

However even with the bottle cached, running HOMEBREW_NO_AUTO_UPDATE=1 brew upgrade pyenv doesn't succeed, presumably because the updated formula is not cached. The command yields:

Error: pyenv 1.1.5 already installed

I added caching of the formula:

cache:
  directories:
    - "$HOME/Library/Caches/Homebrew"
    - /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core

and implemented the following algorithm:

Run HOMEBREW_NO_AUTO_UPDATE=1 brew upgrade pyenv to upgrade pyenv using the latest cached formula/bottle.
Test if pyenv >= 1.2.4 is installed. If the pyenv 1.2.4 formula and bottle were cached, it will be installed in step (1) without brew update having been run.
If pyenv >= 1.2.4 was not cached, then run brew upgrade pyenv and save the cached bottle and formula.

and the following script:

function ver { printf "%03d%03d%03d" $(echo "$1" | tr '.' ' '); }
HOMEBREW_NO_AUTO_UPDATE=1 brew upgrade pyenv &>/dev/null
# Update brew and upgrade pyenv if necessary
if [ $(ver $(pyenv --version | cut -d ' ' -f2)) -lt $(ver 1.2.4) ]; then
  brew upgrade pyenv &>/dev/null
fi

However, I question whether it's a good idea to cache /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core. The examples in the Travis CI documentation are always caching directories under $HOME. It seems like I might experience problems when Travis updates the image. Does Travis CI delete the branch caches after the image is updated? If not, it seems like I would need to do that manually.

Is there a better way? It seems like Travis CI should offer an easy way to cache Brew formula and bottles.

Should I not bother with Brew and just install pyenv from the source?

RjOllos RjOllos · Accepted Answer · 2018-05-09T16:17:52

I cached the brew formula under $HOME by copying the file on the first build:

cache:
  directories:
    - "$HOME/Library/Caches/Homebrew"

[...]

before_install:
  - |
    if [ "$TRAVIS_OS_NAME" = osx ]; then
      # Pyenv >= 1.2.4 is needed with Python 2.6
      if [[ "$pyver" == 2.6 ]]; then
        # Upgrade pyenv and save formula/bottle in cache.
        formula=$(brew --repository)/Library/Taps/homebrew/homebrew-core/Formula/pyenv.rb
        formula_cached=$(brew --cache)/pyenv.rb
        if [[ ! -f $formula_cached ]]; then
          brew upgrade pyenv &>/dev/null
          cp $formula $formula_cached
        else
          HOMEBREW_NO_AUTO_UPDATE=1 brew upgrade $formula_cached
        fi
      fi

Caching Homebrew bottles and formulas on Travis CI

1 Answers