Boosting Travis CI: Python and Buildout caching

Read how to speedup buildout based TravisCI builds, in our case down from 15min to 3min.
Boosting Travis CI: Python and Buildout caching

zc.buildout is Pythons swiss knife to build complex enviroments. MacGywer would love it. Travis CI together with GitHub is a wonderful service for OpenSource projects to do collaborative development hand in hand with Test Driven Development and Continious Integration.

But complex Python builds are taking its time - mainly because of the long list of dependencies and bunch of downloads. It is boring to wait 15minutes for a test

So it was for collective.solr, a package that integrates the excellent Solr open source search platform, (written in Java, from the Apache Lucene project) with the Plone Enterprise CMS. Additional to the complex Plone, it downloads and configures a complete Solr for the test environment.

Since a while Travis CI offers caching on its container based infrastructure.

Using it with buildout is easy once ones knows how.

  1. The file .travis.yaml configures Travis CI, open it.
  2. set language: python if you not already have
  3. an important setting is sudo = false which switches explicit to container based infrastructure. This is default for projects created at Travis CI before 2015-01-01, but explicit is better than implicit!
  4. next the caching is defined. We enable also pip caching. This looks like so
      pip: true
        - $HOME/buildout-cache
  5. in order to create our caching directories a before-install step is needed. In this step we install buildout too. Note: there is no need to use the old and busted any longer (except old and busted Plone builds, since 4.3 at least it will work).
      - mkdir -p $HOME/buildout-cache/{eggs,downloads}
      - virtualenv .
      - bin/pip install --upgrade pip setuptools zc.buildout
  6. Next we need to tweak file travis.cfg. This is the buildout configuration file used for travis. Under section [buildout] add the lines:

    eggs-directory = /home/travis/buildout-cache/eggs
    download-cache = /home/travis/buildout-cache/downloads
    Note, the $HOME environment variable is not available as buildout variable, so we need to set this fixed to /home/travis - Travis CI can not guarantee that this will stick for all future. So if there is a way to access environment variables before buildout runs the parts please let me know.

The second time Travis CI builds the project it took about 3 minutes instead of 15!
The full files as we use it for collective.solr:


language: python
# with next we get on container based infrastructure, this enables caching
sudo: false
  - 2.7
  pip: true
    - $HOME/buildout-cache
  - mkdir -p $HOME/buildout-cache/{eggs,downloads}
  - virtualenv .
  - bin/pip install --upgrade pip setuptools zc.buildout
  - sed -ie "s#plone-x.x.x.cfg#plone-$PLONE_VERSION.cfg#" travis.cfg
  - sed -ie "s#solr-x.x.x.cfg#solr-$SOLR_VERSION.cfg#" travis.cfg
  - bin/buildout -N -t 20 -c travis.cfg
  - bin/code-analysis
  - bin/test
  - pip install -q coveralls
  - coveralls


extends =
parts +=

# caches, see also .travis.yaml
# one should not depend on '/home/travis' but it seems stable in containers.
eggs-directory = /home/travis/buildout-cache/eggs
download-cache = /home/travis/buildout-cache/downloads

recipe = plone.recipe.codeanalysis
pre-commit-hook = False
# return-status-codes = True

We may enhance this, so you can always look at its current state at github/collective/collective.solr.

image by "Albert Einstein - Start 1" by DLR German Aerospace Center at Flickr