how to prevent gitlab ci from downloading sbt every time?

Question

We have a play2/scala application which we are building with gitlab ci.

Our .gitlab-ci.yml (at least the important part) looks as follows:

Answer 1

If you don't want to use custom made images, the best solution is to use Gitlab CI's caching mechanism.

It's a little hard to get it right, but this blog post describes how to do it for SBT.

Example .gitlab-ci.yml

Quoted from the blog post, minor errors corrected by myself:

# some parts originally from https://github.com/randm-ch/units-of-information/blob/master/.gitlab-ci.yml

image: "hseeberger/scala-sbt"

variables:
  SBT_VERSION: "0.13.9"
  SBT_OPTS: "-Dsbt.global.base=sbt-cache/.sbtboot -Dsbt.boot.directory=sbt-cache/.boot -Dsbt.ivy.home=sbt-cache/.ivy"

cache:
  key: "$CI_BUILD_REF_NAME" # contains either the branch or the tag, so it's caching per branch
  untracked: true
  paths:
    - "sbt-cache/.ivy/cache"
    - "sbt-cache/.boot"
    - "sbt-cache/.sbtboot"
    - "sbt-cache/target"

stages:
  - test

test:
  script:
    - sbt test

Second example, also including apt-get caching

This is what I used for my project, usable for more general use cases and Docker images:

image: java:8

stages:
  - test

variables:
  SBT_VERSION: "0.13.9"
  SBT_OPTS: "-Dsbt.global.base=sbt-cache/.sbtboot -Dsbt.boot.directory=sbt-cache/.boot -Dsbt.ivy.home=sbt-cache/.ivy"
  SBT_CACHE_DIR: "sbt-cache/.ivy/cache"

cache:
  key: "$CI_BUILD_REF_NAME" # contains either the branch or the tag, so it's caching per branch
  untracked: true
  paths:
    - "apt-cache/"
    - "sbt-cache/.ivy/cache"
    - "sbt-cache/.boot"
    - "sbt-cache/.sbtboot"
    - "sbt-cache/target"

before_script:
  - export APT_CACHE_DIR=`pwd`/apt-cache
  - mkdir -pv $APT_CACHE_DIR
  - ls $APT_CACHE_DIR || echo "no apt-cache dir found"
  - apt-get -o dir::cache::archives=$APT_CACHE_DIR update -y
  - apt-get -o dir::cache::archives=$APT_CACHE_DIR install apt-transport-https -y
  # Install SBT
  - mkdir -pv $SBT_CACHE_DIR
  - ls $SBT_CACHE_DIR || echo "no ivy2 cache fir found"
  - echo "deb http://dl.bintray.com/sbt/debian /" | tee -a /etc/apt/sources.list.d/sbt.list
  - apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 642AC823
  - apt-get -o dir::cache::archives=$APT_CACHE_DIR update -y
  - apt-get -o dir::cache::archives=$APT_CACHE_DIR install sbt -y
  - sbt -v sbtVersion

test:
  stage: test
  script:
     - sbt -v sbtVersion

Answer 2

There are 4 things you can do:

1. own docker image
2. cache
3. artifacts
4. use own caching solution

The best solution will be a mix of them all.

1. You can build your own docker image with all the dependencies you need, it's the fastest solution as it won't have to download everything but it will introduce another piece of the puzzle you need to take care of. You can use the in-built gitlab repository for storing it and have gitlab build it and then use it.
2. You can use cache in Gitlab CI jobs so it won't have to download everything all the time. The default cache for sbt seems to be ~/.ivy2 so add

cache:
  paths:
    - ~/.ivy2/

As the first lines in your gitlab file to use the caches for each stage.

3. Define the target directory as the artifact to pass it between builds and so you don't have to build it on each stage.

artifacts:
  paths:
    - target/

4. You can store the files you need to cache in your own s3/minio/nfs if the options supplied by gitlab are not enough. This will be a very custom solution so you will have to find your own way around it.