Gitlabs artifact of one project used in further projects

Question

Question

• What is the best way to carry artifacts (jar, class, war) among projects when using docker containers in CI phase.

Let me explain my issue in details, please don't stop the reading... =)

Gitlabs project1

• unit tests
• etc...
• package

Gitlabs project2

• unit test
• etc...
• build (failing)
  • here I need one artifact (jar) generated in project1

Current scenario / comments

• I'm using dockers so in each .gitlab-ci.yml I'll have independent containers
• All is working fine in project1
• If I use "shell" instead of dockers in my .gitlab-ci.yml I can keep the jar file from the project1 in the disk and use when project2 kicks the build
• Today my trigger on call project2 when project1 finish is working nicely
• My artifact is not an RPM so I'll not add into my repo

Possible solutions

• I can commit the artifact of project1 and checkout when need to build project2
• I need to study if cache feature from gitlabs is designed for this purpose (gitlab 8.2.1, How to use cache in .gitlab-ci.yml)

Answer 1

Hello you must take a look at a script named get-last-successful-build-artifact.sh and developed by morph027.

https://gitlab.com/morph027/gitlab-ci-helpers

This script allow to download an artifact and unzip it in the project root. It use Gitlab API to retrieve latest successful build and download corresponding artifact. You can combine multiple artifacts and unzip wherever you want just by updating the script a little.

I'm also currently starting a PHP library to handle build artifacts but it's in a very early stage and tied with laravel for the moment.

For the moment there is no easy way to handle artifact usage between projects, you must build your own using that tools.

I think using shell executor is not the right solution, it's very dangerous because you can't verify the file on the server used during the build !

Hope this help :)

Answer 2

In GitLab silver and premium, there is the $CI_JOB_TOKEN available, which allows the following .gitlab-ci.yaml snippet:

build_submodule:
  image: debian
  stage: test
  script:
  - apt update && apt install -y unzip
  - curl --location --output artifacts.zip "https://gitlab.example.com/api/v4/projects/1/jobs/artifacts/master/download?job=test&job_token=$CI_JOB_TOKEN"
  - unzip artifacts.zip
  only:
  - tags

However, if you do not have silver or higher gitlab subscriptions, but rely on free tiers, it is also possible to use the API and pipeline triggers.

Let's assume we have project A building app.jar which is needed by project B.

First, you will need an API Token. Go to your settings to create one, then store it as a variable in project B. In my example it's GITLAB_API_TOKEN.

In the CI / CD settings of project B add a new trigger, for example "Project A built". This will give you a token which you can copy. Open project A's .gitlab-ci.yaml and copy the trigger_build: section from project B's CI / CD settings trigger section.

Project A:

trigger_build:
  stage: deploy
  script:
    - "curl -X POST -F token=TOKEN -F ref=REF_NAME https://gitlab.example.com/api/v4/projects/${PROJECT_B_ID}/trigger/pipeline"

Replace TOKEN with that token (better, store it as a variable in project A -- then you will need to make it token=${TRIGGER_TOKEN_PROJECT_B} or something), and replace REF_NAME with your branch (e.g. master).

Then, in project B, we can write a section which only builds on triggers and retrieves the artifacts.

Project B:

download:
  stage: deploy
  only:
    - triggers
  script:
    - "curl -O --header 'PRIVATE-TOKEN: ${GITLAB_API_TOKEN}' https://gitlab.example.com/api/v4/projects/${PROJECT_A_ID}/jobs/${REMOTE_JOB_ID}/artifacts/${REMOTE_FILENAME}"

If you know the artifact path, then you can replace ${REMOTE_FILENAME} with it, for example build/app.jar. The project ID can be found in the CI / CD settings.

I extended the script in project A to pass the remaining information as documented in the trigger settings section:

Add variables[VARIABLE]=VALUE to an API request. Variable values can be used to distinguish between triggered pipelines and normal pipelines.

So the trigger passes the REMOTE_JOB_ID and the REMOTE_FILENAME, but of course you can modify this as you need it:

curl -X POST \
     -F token=TOKEN \
     -F ref=REF_NAME \
     -F "variables[REMOTE_FILENAME]=build/app.jar" \
     -F "variables[REMOTE_JOB_ID]=${CI_JOB_ID}" \
     https://gitlab.example.com/api/v4/projects/${PROJECT_B_ID}/trigger/pipeline

Answer 3

Cool, found my snippet being referenced here ;)

is it possible to use get-last-successful-build-artifact.sh without private-token (in a world-readable repository)? e.g. to share an artifact download command w/o exposing your token

Yes, just add it as a secret variable in project settings -> pipelines -> secret variables.

Answer 4

As of this writing artifacts cannot be shared across project only within the pipeline. See https://docs.gitlab.com/ee/ci/yaml/README.html#artifacts

However there is an open feature to enable this facility which is not yet implemented. https://gitlab.com/gitlab-org/gitlab-ce/issues/14728