Jenkins Pipeline NotSerializableException: groovy.json.internal.LazyMap

Question

Solved: Thanks to below answer from S.Richmond. I needed to unset all stored maps of the groovy.json.internal.LazyMap type whi

Answer 1

I found more easy way in off docs for Jenkins pipeline

Work example

import groovy.json.JsonSlurperClassic 


@NonCPS
def jsonParse(def json) {
    new groovy.json.JsonSlurperClassic().parseText(json)
}

@NonCPS
def jobs(list) {
    list
        .grep { it.value == true  }
        .collect { [ name : it.key.toString(),
                      branch : it.value.toString() ] }

}

node {
    def params = jsonParse(env.choice_app)
    def forBuild = jobs(params)
}

Due to limitations in Workflow - i.e., JENKINS-26481 - it's not really possible to use Groovy closures or syntax that depends on closures, so you can't > do the Groovy standard of using .collectEntries on a list and generating the steps as values for the resulting entries. You also can't use the standard > Java syntax for For loops - i.e., "for (String s: strings)" - and instead have to use old school counter-based for loops.

Answer 2

Noob mistake on my part. Moved someones code from a old pipeline plugin, jenkins 1.6? to a server running the latest 2.x jenkins.

Failed for this reason: "java.io.NotSerializableException: groovy.lang.IntRange" I kept reading and reading this post multiple times for the above error. Realized: for (num in 1..numSlaves) { IntRange - non-serializable object type.

Rewrote in simple form: for (num = 1; num <= numSlaves; num++)

All is good with the world.

I do not use java or groovy very often.

Thanks guys.

Answer 3

Use JsonSlurperClassic instead.

Since Groovy 2.3 (note: Jenkins 2.7.1 uses Groovy 2.4.7) JsonSlurper returns LazyMap instead of HashMap. This makes new implementation of JsonSlurper not thread safe and not serializable. This makes it unusable outside of @NonDSL functions in pipeline DSL scripts.

However you can fall-back to groovy.json.JsonSlurperClassic which supports old behavior and could be safely used within pipeline scripts.

Example

import groovy.json.JsonSlurperClassic 


@NonCPS
def jsonParse(def json) {
    new groovy.json.JsonSlurperClassic().parseText(json)
}

node('master') {
    def config =  jsonParse(readFile("config.json"))

    def db = config["database"]["address"]
    ...
}    

ps. You still will need to approve JsonSlurperClassic before it could be called.

Answer 4

According to best practices posted on Jenkins blog (Pipeline scalability best practice), it is strongly recommended to use command-line tools or scripts for this kind of work :

Gotcha: especially avoid Pipeline XML or JSON parsing using Groovy’s XmlSlurper and JsonSlurper! Strongly prefer command-line tools or scripts.

i. The Groovy implementations are complex and as a result more brittle in Pipeline use.

ii. XmlSlurper and JsonSlurper can carry a high memory and CPU cost in pipelines

iii. xmllint and xmlstartlet are command-line tools offering XML extraction via xpath

iv. jq offers the same functionality for JSON

v. These extraction tools may be coupled to curl or wget for fetching information from an HTTP API

Thus, it explains why most solutions proposed on this page are blocked by default by Jenkins security script plugin's sandbox.

The language philosophy of Groovy is closer to Bash than Python or Java. Also, it means it's not natural to do complex and heavy work in native Groovy.

Given that, I personally decided to use the following :

sh('jq <filters_and_options> file.json')

See jq Manual and Select objects with jq stackoverflow post for more help.

This is a bit counter intuitive because Groovy provides many generic methods that are not in the default whitelist.

If you decide to use Groovy language anyway for most of your work, with sandbox enabled and clean (which is not easy because not natural), I recommend you to check the whitelists for your security script plugin's version to know what are your possibilities : Script security plugin whitelists

Answer 5

I ran into this myself today and through some bruteforce I've figured out both how to resolve it and potentially why.

Probably best to start with the why:

Jenkins has a paradigm where all jobs can be interrupted, paused and resumable through server reboots. To achieve this the pipeline and its data must be fully serializable - IE it needs to be able to save the state of everything. Similarly, it needs to be able to serialize the state of global variables between nodes and sub-jobs in the build, which is what I think is happening for you and I and why it only occurs if you add that additional build step.

For whatever reason JSONObject's aren't serializable by default. I'm not a Java dev so I cannot say much more on the topic sadly. There are plenty of answers out there about how one may fix this properly though I do not know how applicable they are to Groovy and Jenkins. See this post for a little more info.

How you fix it:

If you know how, you can possibly make the JSONObject serializable somehow. Otherwise you can resolve it by ensuring no global variables are of that type.

Try unsetting your object var or wrapping it in a method so its scope isn't node global.

Answer 6

A slightly more generalized form of the answer from @mkobit which would allow decoding of arrays as well as maps would be:

import groovy.json.JsonSlurper

@NonCPS
def parseJsonText(String json) {
  def object = new JsonSlurper().parseText(json)
  if(object instanceof groovy.json.internal.LazyMap) {
      return new HashMap<>(object)
  }
  return object
}

NOTE: Be aware that this will only convert the top level LazyMap object to a HashMap. Any nested LazyMap objects will still be there and continue to cause issues with Jenkins.