Currying groovy CPS closure for parallel execution

Question

We do some dynamic creation of parallel steps in some of our jobs. Thanks to this thread I found how to dynamically create the map with parameters for use in the parallel step.

However now I wanted to reuse parts of the code which is used to create those parallel steps. For this I feel that I would need to curry the closures.

However currying seems not to work correctly. Referencing the loop variable (valueCopy) inside the closure does the right thing (as mentioned here) but currying does not do what I would expect.

Am I doing something wrong, is this just not (yet) supported, are there any workarounds? Is this probably a bug in Jenkins pipeline?

Hope anybody has a clue why this doesn't work and/or how to get it working.

Jenkins: LTS (2.32.1) & latest plugin updates as of 2017/01/19.

Solution:

After upgrading to Pipeline: Groovy plugin version 2.40 eveything is working as expected now.

Pipeline script executed:

def echoSome(val) {
    echo val
}

def buildClosures() {
    def someList = ["1", "2", "3"]
    def closures = [:]
    for (value in someList) {
        final valueCopy = value

        closures[value] = {val ->
                echo valueCopy.toString()
                echo val.toString()
            }.curry(value)
    }
    closures
}

parallel buildClosures()

Output:

[Pipeline] parallel
[Pipeline] [1] { (Branch: 1)
[Pipeline] [2] { (Branch: 2)
[Pipeline] [3] { (Branch: 3)
[Pipeline] [1] echo
[1] 1
[Pipeline] [1] echo
[1] 3
[Pipeline] [1] }
[Pipeline] [2] echo
[2] 2
[Pipeline] [2] echo
[2] 3
[Pipeline] [2] }
[Pipeline] [3] echo
[3] 3
[Pipeline] [3] echo
[3] 3
[Pipeline] [3] }
[Pipeline] // parallel
[Pipeline] End of Pipeline
Finished: SUCCESS

Expected Output:

[Pipeline] parallel
[Pipeline] [1] { (Branch: 1)
[Pipeline] [2] { (Branch: 2)
[Pipeline] [3] { (Branch: 3)
[Pipeline] [1] echo
[1] 1
[Pipeline] [1] echo
[1] 1
[Pipeline] [1] }
[Pipeline] [2] echo
[2] 2
[Pipeline] [2] echo
[2] 2
[Pipeline] [2] }
[Pipeline] [3] echo
[3] 3
[Pipeline] [3] echo
[3] 3
[Pipeline] [3] }
[Pipeline] // parallel
[Pipeline] End of Pipeline
Finished: SUCCESS

Answer 1

I'm not sure if it's the currying, or the for loop, but this function needs to be marked as NonCPS as described here: https://github.com/jenkinsci/pipeline-examples/blob/master/docs/BEST_PRACTICES.md#groovy-gotchas

Essentially, do this:

@NonCPS
def buildClosures() {
    def someList = ["1", "2", "3"]
    def closures = [:]
    for (value in someList) {
        final valueCopy = value

        closures[value] = {val ->
                echo valueCopy.toString()
                echo val.toString()
            }.curry(value)
    }
    closures
}

I think it's your for loop, but regardless, anytime you don't use classic "C Style" loops, you'll need to mark your function as NonCPS.

Answer 2

This seems to be a limitation of either the Groovy language or the Jenkins groovy runtime, I'm not sure which, but it's worth noting their examples do exactly as you have done, declaring a new variable for each iteration of the loop.

They have commented their example

// fresh variable per iteration; i will be mutated

I don't think that using C-style loops will remove this limitation and currying (which would be required for this use case) does not resolve the issue either. Clumsy, but easy enough to work around.

https://github.com/jenkinsci/pipeline-plugin/blob/master/TUTORIAL.md#creating-multiple-threads

Answer 3

Found that with the latest Pipeline: Groovy plugin (2.40) combined with at least Jenkins Version 2.60.3 (works although the plug-ins homepage states that you need at least Jenkins 2.73.3) everything works as expected.