詹金斯管道 NotSerializableException: groovy.json.internal.LazyMap

Jenkins Pipeline NotSerializableException: groovy.json.internal.LazyMap

提问by Sunvic

Solved: Thanks to below answerfrom S.Richmond. I needed to unset allstored maps of the groovy.json.internal.LazyMaptype which meant nullifying the variables envServersand objectafter use.

已解决：感谢S.Richmond 的以下回答。我需要来取消所有存储的地图groovy.json.internal.LazyMap，这意味着抵消变量类型envServers和object使用后。

Additional: People searching for this error might be interested to use the Jenkins pipeline step readJSONinstead - find more info here.

附加：搜索此错误的人可能有兴趣改用 Jenkins 管道步骤readJSON-在此处查找更多信息。

---

---

I am trying to use Jenkins Pipeline to take input from the user which is passed to the job as json string. Pipeline then parses this using the slurper and I pick out the important information. It will then use that information to run 1 job multiple times in parallel with differeing job parameters.

我正在尝试使用 Jenkins Pipeline 从用户那里获取输入，该输入作为 json 字符串传递给作业。管道然后使用 slurper 解析它，我挑选出重要的信息。然后它将使用该信息以不同的作业参数并行运行 1 个作业。

Up until I add the code below "## Error when below here is added"the script will run fine. Even the code below that point will run on its own. But when combined I get the below error.

直到我"## Error when below here is added"在脚本下面添加代码才能正常运行。即使低于该点的代码也会自行运行。但是当合并时，我得到以下错误。

I should note that the triggered job is called and does run succesfully but the below error occurs and fails the main job. Because of this the main job does not wait for the return of the triggered job. I couldtry/catch around the build job:however I want the main job to wait for the triggered job to finish.

我应该注意到触发的作业被调用并成功运行，但发生以下错误并使主作业失败。因此，主作业不会等待触发作业的返回。我可以尝试/抓住build job:但我希望主要工作等待触发的工作完成。

Can anyone assist here? If you need anymore information let me know.

有人可以在这里提供帮助吗？如果您需要更多信息，请告诉我。

Cheers

干杯

def slurpJSON() {
return new groovy.json.JsonSlurper().parseText(BUILD_CHOICES);
}

node {
  stage 'Prepare';
  echo 'Loading choices as build properties';
  def object = slurpJSON();

  def serverChoices = [];
  def serverChoicesStr = '';

  for (env in object) {
     envName = env.name;
     envServers = env.servers;

     for (server in envServers) {
        if (server.Select) {
            serverChoicesStr += server.Server;
            serverChoicesStr += ',';
        }
     }
  }
  serverChoicesStr = serverChoicesStr[0..-2];

  println("Server choices: " + serverChoicesStr);

  ## Error when below here is added

  stage 'Jobs'
  build job: 'Dummy Start App', parameters: [[$class: 'StringParameterValue', name: 'SERVER_NAME', value: 'TestServer'], [$class: 'StringParameterValue', name: 'SERVER_DOMAIN', value: 'domain.uk'], [$class: 'StringParameterValue', name: 'APP', value: 'application1']]

}

Error:

错误：

java.io.NotSerializableException: groovy.json.internal.LazyMap
    at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:860)
    at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:569)
    at org.jboss.marshalling.river.BlockMarshaller.doWriteObject(BlockMarshaller.java:65)
    at org.jboss.marshalling.river.BlockMarshaller.writeObject(BlockMarshaller.java:56)
    at org.jboss.marshalling.MarshallerObjectOutputStream.writeObjectOverride(MarshallerObjectOutputStream.java:50)
    at org.jboss.marshalling.river.RiverObjectOutputStream.writeObjectOverride(RiverObjectOutputStream.java:179)
    at java.io.ObjectOutputStream.writeObject(Unknown Source)
    at java.util.LinkedHashMap.internalWriteEntries(Unknown Source)
    at java.util.HashMap.writeObject(Unknown Source)
...
...
Caused by: an exception which occurred:
    in field delegate
    in field closures
    in object org.jenkinsci.plugins.workflow.cps.CpsThreadGroup@5288c

采纳答案by S.Richmond

I ran into this myself today and through some bruteforce I've figured out both how to resolve it and potentially why.

我今天自己遇到了这个问题，通过一些蛮力我已经想出了如何解决它以及可能的原因。

Probably best to start with the why:

可能最好从原因开始：

Jenkins has a paradigm where all jobs can be interrupted, paused and resumable through server reboots. To achieve this the pipeline and its data must be fully serializable - IE it needs to be able to save the state of everything. Similarly, it needs to be able to serialize the state of global variables between nodes and sub-jobs in the build, which is what I think is happening for you and I and why it only occurs if you add that additional build step.

Jenkins 有一个范例，所有作业都可以通过服务器重启来中断、暂停和恢复。为了实现这一点，管道及其数据必须是完全可序列化的——IE 它需要能够保存一切的状态。同样，它需要能够在构建中的节点和子作业之间序列化全局变量的状态，这就是我认为正在发生的事情，以及为什么只有在添加额外的构建步骤时才会发生这种情况。

For whatever reason JSONObject's aren't serializable by default. I'm not a Java dev so I cannot say much more on the topic sadly. There are plenty of answers out there about how one may fix this properly though I do not know how applicable they are to Groovy and Jenkins. See this postfor a little more info.

无论出于何种原因，默认情况下 JSONObject 都不可序列化。我不是 Java 开发人员，所以遗憾地不能就这个话题说更多。关于如何正确解决这个问题，有很多答案，尽管我不知道它们对 Groovy 和 Jenkins 有多适用。有关更多信息，请参阅此帖子。

How you fix it:

你如何修复它：

If you know how, you can possibly make the JSONObject serializable somehow. Otherwise you can resolve it by ensuring no global variables are of that type.

如果你知道如何，你可以以某种方式使 JSONObject 可序列化。否则，您可以通过确保没有该类型的全局变量来解决它。

Try unsetting your objectvar or wrapping it in a method so its scope isn't node global.

尝试取消设置您的objectvar 或将其包装在一个方法中，使其范围不是节点全局的。

回答by luka5z

Use JsonSlurperClassicinstead.

使用JsonSlurperClassic来代替。

Since Groovy 2.3 (note: Jenkins 2.7.1 uses Groovy 2.4.7) JsonSlurperreturns LazyMapinstead of HashMap. This makes new implementation of JsonSlurpernotthread safe and notserializable. This makes it unusable outside of @NonDSL functions in pipeline DSL scripts.

由于 Groovy 2.3（注意：Jenkins 2.7.1 使用 Groovy 2.4.7）JsonSlurper返回LazyMap而不是HashMap. 这使得JsonSlurper非线程安全和不可序列化的新实现。这使得它在管道 DSL 脚本中的 @NonDSL 函数之外无法使用。

However you can fall-back to groovy.json.JsonSlurperClassicwhich supports old behaviorand could be safely used within pipeline scripts.

但是，您可以回退到groovy.json.JsonSlurperClassic支持旧行为并且可以在管道脚本中安全使用的方法。

Example

例子

import groovy.json.JsonSlurperClassic 


@NonCPS
def jsonParse(def json) {
    new groovy.json.JsonSlurperClassic().parseText(json)
}

node('master') {
    def config =  jsonParse(readFile("config.json"))

    def db = config["database"]["address"]
    ...
}    

ps. You still will need to approve JsonSlurperClassicbefore it could be called.

附：您仍然需要批准JsonSlurperClassic才能调用它。

回答by mkobit

EDIT:As pointed out by @Sunvicin the comments, the below solution does not work as-is for JSON Arrays.

编辑：正如@Sunvic在评论中指出的那样，以下解决方案不适用于 JSON 数组。

I dealt with this by using JsonSlurperand then creating a new HashMapfrom the lazy results. HashMapis Serializable.

我通过使用JsonSlurper然后HashMap从懒惰的结果创建一个新的来处理这个问题。HashMap是Serializable。

I believe that this required whitelisting of both the new HashMap(Map)and the JsonSlurper.

我相信这需要new HashMap(Map)将JsonSlurper.

@NonCPS
def parseJsonText(String jsonText) {
  final slurper = new JsonSlurper()
  return new HashMap<>(slurper.parseText(jsonText))
}

Overall, I would recommend just using the Pipeline Utility Stepsplugin, as it has a readJSONstepthat can support either files in the workspace or text.

总的来说，我建议只使用Pipeline Utility Steps插件，因为它有一个readJSON步骤可以支持工作区中的文件或文本。

回答by TomDotTom

A slightly more generalized form of the answer from @mkobit which would allow decoding of arrays as well as maps would be:

来自@mkobit 的答案的稍微更通用的形式将允许对数组和映射进行解码：

import groovy.json.JsonSlurper

@NonCPS
def parseJsonText(String json) {
  def object = new JsonSlurper().parseText(json)
  if(object instanceof groovy.json.internal.LazyMap) {
      return new HashMap<>(object)
  }
  return object
}

NOTE: Be aware that this will only convert the top level LazyMap object to a HashMap. Any nested LazyMap objects will still be there and continue to cause issues with Jenkins.

注意：请注意，这只会将顶级 LazyMap 对象转换为 HashMap。任何嵌套的 LazyMap 对象仍将存在并继续导致 Jenkins 出现问题。

回答by Regnoult

I want to upvote one of the answer: I would recommend just using the Pipeline Utility Steps plugin, as it has a readJSON step that can support either files in the workspace or text: https://jenkins.io/doc/pipeline/steps/pipeline-utility-steps/#readjson-read-json-from-files-in-the-workspace

我想支持其中一个答案：我建议只使用 Pipeline Utility Steps 插件，因为它有一个 readJSON 步骤，可以支持工作区中的文件或文本：https://jenkins.io/doc/pipeline/steps /pipeline-utility-steps/#readjson-read-json-from-files-in-the-workspace

script{
  def foo_json = sh(returnStdout:true, script: "aws --output json XXX").trim()
  def foo = readJSON text: foo_json
}

This does NOT require any whitelisting or additional stuff.

这不需要任何白名单或额外的东西。

回答by Nils El-Himoud

This is the detailed answer that was asked for.

这是要求的详细答案。

The unset worked for me:

未设置对我有用：

String res = sh(script: "curl --header 'X-Vault-Token: ${token}' --request POST --data '${payload}' ${url}", returnStdout: true)
def response = new JsonSlurper().parseText(res)
String value1 = response.data.value1
String value2 = response.data.value2

// unset response because it's not serializable and Jenkins throws NotSerializableException.
response = null

I read the values from the parsed response and when I don't need the object anymore I unset it.

我从解析的响应中读取值，当我不再需要该对象时，我会取消设置它。

回答by vhamon

According to best practices posted on Jenkins blog (Pipeline scalability best practice), it is strongly recommended to use command-line tools or scriptsfor this kind of work :

根据 Jenkins 博客上发布的最佳实践（管道可扩展性最佳实践），强烈建议使用命令行工具或脚本进行此类工作：

Gotcha: especially avoid Pipeline XML or JSON parsing using Groovy's XmlSlurper and JsonSlurper! Strongly prefer command-line tools or scripts.

i. The Groovy implementations are complex and as a result more brittle in Pipeline use.

ii. XmlSlurper and JsonSlurper can carry a high memory and CPU cost in pipelines

iii. xmllint and xmlstartlet are command-line tools offering XML extraction via xpath

iv. jq offers the same functionality for JSON

v. These extraction tools may be coupled to curl or wget for fetching information from an HTTP API

问题：尤其要避免使用 Groovy 的 XmlSlurper 和 JsonSlurper 解析流水线 XML 或 JSON！强烈喜欢命令行工具或脚本。

一世。Groovy 实现很复杂，因此在 Pipeline 的使用中更加脆弱。

ii. XmlSlurper 和 JsonSlurper 可以在管道中承载高内存和 CPU 成本

三、xmllint 和 xmlstartlet 是通过 xpath 提供 XML 提取的命令行工具

四、jq 为 JSON 提供相同的功能

v. 这些提取工具可以耦合到 curl 或 wget 以从 HTTP API 获取信息

Thus, it explains why most solutions proposed on this page are blocked by default by Jenkins security script plugin's sandbox.

因此，它解释了为什么在此页面上提出的大多数解决方案默认被 Jenkins 安全脚本插件的沙箱阻止。

The language philosophy of Groovy is closer to Bash than Python or Java. Also, it means it's not natural to do complex and heavy work in native Groovy.

Groovy 的语言哲学比 Python 或 Java 更接近 Bash。此外，这意味着在原生 Groovy 中进行复杂而繁重的工作并不自然。

Given that, I personally decided to use the following :

鉴于此，我个人决定使用以下内容：

sh('jq <filters_and_options> file.json')

See jq Manualand Select objects with jq stackoverflow postfor more help.

有关更多帮助，请参阅jq Manual和Select objects with jq stackoverflow post。

This is a bit counter intuitive because Groovy provides many generic methods that are not in the default whitelist.

这有点违反直觉，因为 Groovy 提供了许多不在默认白名单中的通用方法。

If you decide to use Groovy language anyway for most of your work, with sandbox enabled and clean (which is not easy because not natural), I recommend you to check the whitelists for your security script plugin's version to know what are your possibilities : Script security plugin whitelists

如果您决定在大部分工作中使用 Groovy 语言，启用沙箱并保持干净（这并不容易，因为不自然），我建议您检查安全脚本插件版本的白名单，以了解您的可能性：脚本安全插件白名单

回答by Marcin P?onka

The way pipeline plugin has been implemented has quite serious implications for non-trivial Groovy code. This link explains how to avoid possible problems: https://github.com/jenkinsci/pipeline-plugin/blob/master/TUTORIAL.md#serializing-local-variables

管道插件的实现方式对非平凡的 Groovy 代码具有相当严重的影响。此链接解释了如何避免可能出现的问题：https: //github.com/jenkinsci/pipeline-plugin/blob/master/TUTORIAL.md#serializing-local-variables

In your specific case I'd consider adding @NonCPSannotation to slurpJSONand returning map-of-maps instead of JSON object. Not only the code looks cleaner, but it's also more efficient, especially if that JSON is complex.

在您的特定情况下，我会考虑添加@NonCPS注释slurpJSON并返回 map-of-maps 而不是 JSON 对象。不仅代码看起来更简洁，而且效率也更高，尤其是当 JSON 很复杂时。

回答by Stevel

The other ideas in this post were helpful, but not quite all I was looking for - so I extracted the parts that fit my need and added some of my own magix...

这篇文章中的其他想法很有帮助，但并不是我想要的全部 - 所以我提取了适合我需要的部分并添加了一些我自己的魔法......

def jsonSlurpLaxWithoutSerializationTroubles(String jsonText)
{
    return new JsonSlurperClassic().parseText(
        new JsonBuilder(
            new JsonSlurper()
                .setType(JsonParserType.LAX)
                .parseText(jsonText)
        )
        .toString()
    )
}

Yes, as I noted in my own git commit of the code, "Wildly-ineffecient, but tiny coefficient: JSON slurp solution"(which I'm okay with for this purpose). The aspects I needed to solve:

是的，正如我在自己的代码 git commit 中所指出的，“效率极低，但系数很小：JSON slurp 解决方案”（我对此表示同意）。我需要解决的方面：

1. Completely get away from the java.io.NotSerializableExceptionproblem, even when the JSON text defines nested containers
2. Work for both map and array containers
3. Support LAX parsing (the most important part, for my situation)
4. Easy to implement (even with the awkward nested constructors that obviate @NonCPS)

1. 完全摆脱这个java.io.NotSerializableException问题，即使 JSON 文本定义了嵌套容器
2. 适用于地图和数组容器
3. 支持 LAX 解析（最重要的部分，就我的情况而言）
4. 易于实现（即使使用笨拙的嵌套构造函数也可以避免@NonCPS）

回答by mpechner

Noob mistake on my part. Moved someones code from a old pipeline plugin, jenkins 1.6? to a server running the latest 2.x jenkins.

我的菜鸟错误。从旧的管道插件 jenkins 1.6 中移动某人的代码？到运行最新 2.x jenkins 的服务器。

Failed for this reason: "java.io.NotSerializableException: groovy.lang.IntRange" I kept reading and reading this post multiple times for the above error. Realized: for (num in 1..numSlaves) { IntRange - non-serializable object type.

由于这个原因而失败：“java.io.NotSerializableException: groovy.lang.IntRange”我多次阅读这篇文章以解决上述错误。实现： for (num in 1..numSlaves) { IntRange - 不可序列化的对象类型。

Rewrote in simple form: for (num = 1; num <= numSlaves; num++)

以简单的形式重写： for (num = 1; num <= numSlaves; num++)

All is good with the world.

世界上一切都好。

I do not use java or groovy very often.

我不经常使用 java 或 groovy。

Thanks guys.

谢谢你们。