1. 技术文章
  2. Jenkins Pipeline NotSerializableException: groovy.json.internal.LazyMap

Jenkins Pipeline NotSerializableException: groovy.json.internal.LazyMap

由 人盡茶涼 提交于 2019-11-27 07:03:09
S.Richmond

I ran into this myself today and through some bruteforce I've figured out both how to resolve it and potentially why.

Probably best to start with the why:

Jenkins has a paradigm where all jobs can be interrupted, paused and resumable through server reboots. To achieve this the pipeline and its data must be fully serializable - IE it needs to be able to save the state of everything. Similarly, it needs to be able to serialize the state of global variables between nodes and sub-jobs in the build, which is what I think is happening for you and I and why it only occurs if you add that additional build step.

For whatever reason JSONObject's aren't serializable by default. I'm not a Java dev so I cannot say much more on the topic sadly. There are plenty of answers out there about how one may fix this properly though I do not know how applicable they are to Groovy and Jenkins. See this post for a little more info.

How you fix it:

If you know how, you can possibly make the JSONObject serializable somehow. Otherwise you can resolve it by ensuring no global variables are of that type.

Try unsetting your object var or wrapping it in a method so its scope isn't node global.

Use JsonSlurperClassic instead.

Since Groovy 2.3 (note: Jenkins 2.7.1 uses Groovy 2.4.7) JsonSlurper returns LazyMap instead of HashMap. This makes new implementation of JsonSlurper not thread safe and not serializable. This makes it unusable outside of @NonDSL functions in pipeline DSL scripts.

However you can fall-back to groovy.json.JsonSlurperClassic which supports old behavior and could be safely used within pipeline scripts.

Example

import groovy.json.JsonSlurperClassic 


@NonCPS
def jsonParse(def json) {
    new groovy.json.JsonSlurperClassic().parseText(json)
}

node('master') {
    def config =  jsonParse(readFile("config.json"))

    def db = config["database"]["address"]
    ...
}

ps. You still will need to approve JsonSlurperClassic before it could be called.

EDIT: As pointed out by @Sunvic in the comments, the below solution does not work as-is for JSON Arrays.

I dealt with this by using JsonSlurper and then creating a new HashMap from the lazy results. HashMap is Serializable.

I believe that this required whitelisting of both the new HashMap(Map) and the JsonSlurper.

@NonCPS
def parseJsonText(String jsonText) {
  final slurper = new JsonSlurper()
  return new HashMap<>(slurper.parseText(jsonText))
}

Overall, I would recommend just using the Pipeline Utility Steps plugin, as it has a readJSON step that can support either files in the workspace or text.

A slightly more generalized form of the answer from @mkobit which would allow decoding of arrays as well as maps would be:

import groovy.json.JsonSlurper

@NonCPS
def parseJsonText(String json) {
  def object = new JsonSlurper().parseText(json)
  if(object instanceof groovy.json.internal.LazyMap) {
      return new HashMap<>(object)
  }
  return object
}

NOTE: Be aware that this will only convert the top level LazyMap object to a HashMap. Any nested LazyMap objects will still be there and continue to cause issues with Jenkins.

This is the detailed answer that was asked for.

The unset worked for me:

String res = sh(script: "curl --header 'X-Vault-Token: ${token}' --request POST --data '${payload}' ${url}", returnStdout: true)
def response = new JsonSlurper().parseText(res)
String value1 = response.data.value1
String value2 = response.data.value2

// unset response because it's not serializable and Jenkins throws NotSerializableException.
response = null

I read the values from the parsed response and when I don't need the object anymore I unset it.

The way pipeline plugin has been implemented has quite serious implications for non-trivial Groovy code. This link explains how to avoid possible problems: https://github.com/jenkinsci/pipeline-plugin/blob/master/TUTORIAL.md#serializing-local-variables

In your specific case I'd consider adding @NonCPS annotation to slurpJSON and returning map-of-maps instead of JSON object. Not only the code looks cleaner, but it's also more efficient, especially if that JSON is complex.

Noob mistake on my part. Moved someones code from a old pipeline plugin, jenkins 1.6? to a server running the latest 2.x jenkins.

Failed for this reason: "java.io.NotSerializableException: groovy.lang.IntRange" I kept reading and reading this post multiple times for the above error. Realized: for (num in 1..numSlaves) { IntRange - non-serializable object type.

Rewrote in simple form: for (num = 1; num <= numSlaves; num++)

All is good with the world.

I do not use java or groovy very often.

Thanks guys.

I want to upvote one of the answer: I would recommend just using the Pipeline Utility Steps plugin, as it has a readJSON step that can support either files in the workspace or text: https://jenkins.io/doc/pipeline/steps/pipeline-utility-steps/#readjson-read-json-from-files-in-the-workspace

script{
  def foo_json = sh(returnStdout:true, script: "aws --output json XXX").trim()
  def foo = readJSON text: foo_json
}

This does NOT require any whitelisting or additional stuff.

I found more easy way in off docs for Jenkins pipeline

Work example

import groovy.json.JsonSlurperClassic 


@NonCPS
def jsonParse(def json) {
    new groovy.json.JsonSlurperClassic().parseText(json)
}

@NonCPS
def jobs(list) {
    list
        .grep { it.value == true  }
        .collect { [ name : it.key.toString(),
                      branch : it.value.toString() ] }

}

node {
    def params = jsonParse(env.choice_app)
    def forBuild = jobs(params)
}

Due to limitations in Workflow - i.e., JENKINS-26481 - it's not really possible to use Groovy closures or syntax that depends on closures, so you can't > do the Groovy standard of using .collectEntries on a list and generating the steps as values for the resulting entries. You also can't use the standard > Java syntax for For loops - i.e., "for (String s: strings)" - and instead have to use old school counter-based for loops.

The other ideas in this post were helpful, but not quite all I was looking for - so I extracted the parts that fit my need and added some of my own magix...

def jsonSlurpLaxWithoutSerializationTroubles(String jsonText)
{
    return new JsonSlurperClassic().parseText(
        new JsonBuilder(
            new JsonSlurper()
                .setType(JsonParserType.LAX)
                .parseText(jsonText)
        )
        .toString()
    )
}

Yes, as I noted in my own git commit of the code, "Wildly-ineffecient, but tiny coefficient: JSON slurp solution" (which I'm okay with for this purpose). The aspects I needed to solve:

  1. Completely get away from the java.io.NotSerializableException problem, even when the JSON text defines nested containers
  2. Work for both map and array containers
  3. Support LAX parsing (the most important part, for my situation)
  4. Easy to implement (even with the awkward nested constructors that obviate @NonCPS)
标签
json
Jenkins
groovy
jenkins-pipeline
易学教程内所有资源均来自网络或用户发布的内容，如有违反法律规定的内容欢迎反馈！
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!