I am facing an issue where I have two stages defined in my pipeline that both are run on the same node and need to be run in the same workspace.
The first of these stages runs on my master node initially, but towards the end of the steps defined has to unstash some files onto a different node.
The second stage then just needs to continue on my master, and relies on some modules that were installed from the first stage.
Here is my pipeline to better explain:
#!groovy
pipeline {
agent { label 'master' }
stages {
stage('Build') { // 1. Running on master in /var/lib/jenkins/workspace/_Pipelines_IACT-Jenkinsfile-UL3RGRZZQD3LOPY2FUEKN5XCY4ZZ6AGJVM24PLTO3OPL54KTJCEQ
steps {
sh '''
npm install
bower install
gulp set-staging-node-env
gulp prepare-staging-files
gulp webpack
'''
stash includes: 'dist/**/*', name: 'builtSources'
stash includes: 'config/**/*', name: 'appConfig'
node('Protractor') { // 2. Running on vnccentos7 in /var/jenkins/workspace/_Pipelines_IACT-Jenkinsfile-UL3RGRZZQD3LOPY2FUEKN5XCY4ZZ6AGJVM24PLTO3OPL54KTJCEQ
dir('/opt/foo/deploy/') {
unstash 'builtSources'
unstash 'appConfig'
}
}
}
}
stage('Unit Tests') {
agent { label 'master' } // 3. Running on master in /var/lib/jenkins/workspace/_Pipelines_IACT-Jenkinsfile-UL3RGRZZQD3LOPY2FUEKN5XCY4ZZ6AGJVM24PLTO3OPL54KTJCEQ@2
steps {
parallel (
"Jasmine": {
sh 'gulp karma-tests-ci'
},
"Mocha": {
sh 'gulp mocha-tests'
}
)
}
}
}
}
As you can see, I have added comments at the start of each stage\node used to show the jenkins output I see for what workspaces get allocated.
The problem I am facing is that the Unit Tests stage fails as it tries to use some node modules it cannot find. These are present in the first workspace that gets created which is where I want this stage to continue using, so not using a new '@2' suffixed workspace.
Is there a way to tell the Jenkins to preserve previously created workspaces in the pipeline?
EDIT
I am guessing as I have specified agent {label:'master'} again in my next stage, this is what cases a new workspace to get created? Should I be using the node approach instead? Would that allow the same workspace to get used?
I actually have tried using node('master'){...} around each of the parallel steps in my Unit Tests stage, but these still use the @2 suffixed workspace rather than the original.
I have seen other threads talking about how you should not re-use the same workspace as you could run into issues with file locks. They suggest instead to archive\unarchive the workspace between steps.
I have also seen approaches where you can store the workspace path in a variable and use this later on which sounds good for my case, but I havent found any declarative syntax samples, only groovy ones.
EDIT 2
I have now tried a few approaches involving saving the allocated workspace from the first stage into a variable and using in a ws(...) directive in the later stages:
pipeline {
agent { label 'master' }
stages {
stage('Build') {
steps {
script {
def workspace = pwd()
}
sh '''
npm install
bower install
gulp set-staging-node-env
gulp prepare-staging-files
gulp webpack
'''
stash includes: 'dist/**/*', name: 'builtSources'
stash includes: 'config/**/*', name: 'appConfig'
node('Protractor') {
dir('/opt/foo/deploy/') {
unstash 'builtSources'
unstash 'appConfig'
}
}
}
}
stage('Unit Tests') {
steps {
parallel (
"Jasmine": {
node('master') {
ws("${workspace}"){
sh 'gulp karma-tests-ci'
}
}
},
"Mocha": {
node('master') {
ws("${workspace}"){
sh 'gulp mocha-tests'
}
}
}
)
}
post {
success {
sh 'gulp combine-coverage-reports'
sh 'gulp clean-lcov'
publishHTML(target: [
allowMissing: false,
alwaysLinkToLastBuild: false,
keepAll: false,
reportDir: 'test/coverage',
reportFiles: 'index.html',
reportName: 'Test Coverage Report'
])
}
}
}
}
}
I did try just removing the second agent declaration from the Unit Tests stage but the stage remained on my Protractor node which I didnt want it to do. So following the answers\comments here I then used node blocks around each of my parallel steps and used the ws blocks as you can see,
The stage fails, and I can see from the logs that its not using the workspace allocated from the first stage (without the @ suffixes on):
[Jasmine] Running on master in /var/lib/jenkins/workspace/_Pipelines_IACT-Jenkinsfile-UL3RGRZZQD3LOPY2FUEKN5XCY4ZZ6AGJVM24PLTO3OPL54KTJCEQ@2
[Pipeline] [Jasmine] {
[Pipeline] [Jasmine] ws
[Jasmine] Running in /var/lib/jenkins/workspace/_Pipelines_IACT-Jenkinsfile-UL3RGRZZQD3LOPY2FUEKN5XCY4ZZ6AGJVM24PLTO3OPL54KTJCEQ@2@2
[Pipeline] [Jasmine] {
[Pipeline] [Jasmine] sh
[Jasmine] [_Pipelines_IACT-Jenkinsfile-UL3RGRZZQD3LOPY2FUEKN5XCY4ZZ6AGJVM24PLTO3OPL54KTJCEQ@2@2] Running shell script
[Jasmine] + gulp karma-tests-ci
[Jasmine] [08:27:01] No gulpfile found
Its even double suffixing with @2, so I'm unsure what its doing now.
Not sure if it fits your use case, but this example script shows how to share the same node/workspace between different stages & containers:
Additionally, if you're running a Docker agent for a specific stage while specifying agent { label 'whatever' } at the top level, you can ensure that this stage will use the same node and workspace as the rest of the Pipeline:
pipeline {
agent {
label 'whatever'
}
stages {
stage('build') {
steps {
sh "./build-artifact.sh"
}
}
stage('test in docker') {
agent {
docker {
image 'ubuntu:16.04'
reuseNode true
}
}
steps {
sh "./run-tests-in-docker.sh"
}
}
}
}
Use dir instead of ws.
ws automatically amend the prefix "@NUMBER" when the workspace to be used was already occupied by other build job.
dir just move current working directory to exactly where you designate.
pipeline {
agent none
environment {
WIN_WORKSPACE = ""
MAC_WORKSPACE = ""
}
stages {
stage("Build") {
parallel {
stage("Build on Windows") {
agent {
label "windows"
}
steps {
script {
WIN_WORKSPACE = WORKSPACE
}
// steps...
}
}
stage("Build on macOS") {
agent {
label "macos"
}
steps {
script {
MAC_WORKSPACE = WORKSPACE
}
// steps...
}
}
}
}
stage("Deploy") {
parallel {
stage("Deploy on Windows") {
agent {
label "windows"
}
steps {
dir(WIN_WORKSPACE) {
// steps...
}
}
}
stage("Deploy on macOS") {
agent {
label "macos"
}
steps {
dir(MAC_WORKSPACE) {
// steps...
}
}
}
}
}
}
}
Works well as you want.
Specify a custom workspace. From the Pipeline syntax: "ws: Allocate workspace"
ws("/usr/local/jenkins/jobs/custom_workspace") {
stage . . .
stage . . .
}
. . does this work ?
For me, I set the global agent to "agent none", then the stages will use the same workspace.
Example 1: Using "agent none", 2 stages use the some workspace
pipeline {
agent none // <- the difference is here
stages {
stage('prep') {
agent { docker { image 'yourdockerimage' }}
steps {
sh 'pwd' // same workspace, without @2
}
}
stage('build') {
agent { label 'master' }
steps {
sh 'pwd' // same workspace, without @2
}
}
}
}
Example 2: Using a global agent, 2 stages use the difference workspaces
pipeline {
agent { label 'master' } // <- the difference is here
stages {
stage('prep') {
agent { docker { image 'yourdockerimage' }}
steps {
sh 'pwd' // different workspace, with @2 appended
}
}
stage('build') {
// no agent specified
steps {
sh 'pwd' // same workspace, without @2
}
}
}
}
Would appreciate an explanation why this happens. My guess is that the global agent of the whole pipeline, whenever specified will use the "original" workspace (without @2), any other step which uses a different agent will need to use the @2 workspace. So if no global agent is used, all stages share the same workspace
I thing using the node syntax will solve it.
in order to be certain I would use a dir scope to set the workspace on your own
The External Workspace Manager Plugin might solve your problem.
[...] it defines the local path to the workspace, and switches to it.
// Basic usage:
def extWorkspace = exwsAllocate diskPoolId: 'diskpool1'
node ('linux') {
exws (extWorkspace) {
scm checkout
sh 'mvn clean install -DskipTests'
}
}
node ('test') {
exws (extWorkspace) {
sh 'mvn test'
}
}
I am glad to say that, Jenkins can share workspace now.
You can specify stages nested within other stages to share workspace.
The new feature is called Sequential Stages.
I will use your original code to implement it.
#!groovy
pipeline {
agent { label 'master' }
stages {
stage('Build') { // 1. Running on master in /var/lib/jenkins/workspace/_Pipelines_IACT-Jenkinsfile-UL3RGRZZQD3LOPY2FUEKN5XCY4ZZ6AGJVM24PLTO3OPL54KTJCEQ
steps {
sh '''
npm install
bower install
gulp set-staging-node-env
gulp prepare-staging-files
gulp webpack
'''
stash includes: 'dist/**/*', name: 'builtSources'
stash includes: 'config/**/*', name: 'appConfig'
node('Protractor') { // 2. Running on vnccentos7 in /var/jenkins/workspace/_Pipelines_IACT-Jenkinsfile-UL3RGRZZQD3LOPY2FUEKN5XCY4ZZ6AGJVM24PLTO3OPL54KTJCEQ
dir('/opt/foo/deploy/') {
unstash 'builtSources'
unstash 'appConfig'
}
}
}
}
stage('Unit Tests') { // 3. Running on master in /var/lib/jenkins/workspace/_Pipelines_IACT-Jenkinsfile-UL3RGRZZQD3LOPY2FUEKN5XCY4ZZ6AGJVM24PLTO3OPL54KTJCEQ@2
parallel {
stage("Unit Tests@") {
agent {label 'master'}
stages {
stage("Jasmine") {
steps {
sh 'gulp karma-tests-ci'
}
}
stage(Mocha") {
steps {
sh 'gulp mocha-tests'
}
}
}
}
}
}
}
}
来源：https://stackoverflow.com/questions/44259039/how-to-re-use-previously-created-workspace-across-stages