使用 jq 从 bash 中管道分隔的键和值创建 JSON

Create JSON using jq from pipe-separated keys and values in bash

提问by michael_65

I am trying to create a json object from a string in bash. The string is as follows.

我正在尝试从 bash 中的字符串创建一个 json 对象。字符串如下。

CONTAINER|CPU%|MEMUSAGE/LIMIT|MEM%|NETI/O|BLOCKI/O|PIDS
nginx_container|0.02%|25.09MiB/15.26GiB|0.16%|0B/0B|22.09MB/4.096kB|0

The output is from docker stats command and my end goal is to publish custom metrics to aws cloudwatch. I would like to format this string as json.

输出来自 docker stats 命令，我的最终目标是将自定义指标发布到 aws cloudwatch。我想将此字符串格式化为 json。

{
    "CONTAINER":"nginx_container",
    "CPU%":"0.02%", 
    ....
}

I have used jq command before and it seems like it should work well in this case but I have not been able to come up with a good solution yet. Other than hardcoding variable names and indexing using sed or awk. Then creating a json from scratch. Any suggestions would be appreciated. Thanks.

我之前使用过 jq 命令，在这种情况下它似乎应该可以正常工作，但我还没有想出一个好的解决方案。除了使用 sed 或 awk 硬编码变量名称和索引。然后从头开始创建一个json。任何建议，将不胜感激。谢谢。

回答by Charles Duffy

Prerequisite

先决条件

For all of the below, it's assumed that your content is in a shell variable named s:

对于以下所有内容，假设您的内容位于名为 的 shell 变量中s：

s='CONTAINER|CPU%|MEMUSAGE/LIMIT|MEM%|NETI/O|BLOCKI/O|PIDS
nginx_container|0.02%|25.09MiB/15.26GiB|0.16%|0B/0B|22.09MB/4.096kB|0'

What (modern jq)

什么（现代 jq）

# thanks to @JeffMercado and @chepner for refinements, see comments
jq -Rn '
( input  | split("|") ) as $keys |
( inputs | split("|") ) as $vals |
[[$keys, $vals] | transpose[] | {key:.[0],value:.[1]}] | from_entries
' <<<"$s"

How (modern jq)

如何（现代 jq）

This requires very new (probably 1.5?) jqto work, and is a dense chunk of code. To break it down:

这需要非常新的（可能是 1.5？）jq才能工作，并且是一段密集的代码。分解：

• Using -nprevents jqfrom reading stdin on its own, leaving the entirety of the input stream available to be read by inputand inputs-- the former to read a single line, and the latter to read all remaining lines. (-R, for raw input, causes textual lines rather than JSON objects to be read).
• With [$keys, $vals] | transpose[], we're generating [key, value]pairs (in Python terms, zipping the two lists).
• With {key:.[0],value:.[1]}, we're making each [key, value]pair into an object of the form {"key": key, "value": value}
• With from_entries, we're combining those pairs into objects containing those keys and values.

• 使用-n防止jq从自身读取标准输入，而使输入流提供给由被读取的全部input和inputs-前者读取单个线，后者读取所有剩余的行。( -R，对于原始输入，会导致读取文本行而不是 JSON 对象)。
• 使用[$keys, $vals] | transpose[]，我们正在生成[key, value]对（用 Python 术语，压缩两个列表）。
• 使用{key:.[0],value:.[1]}，我们将每一[key, value]对组合成一个对象形式{"key": key, "value": value}
• 使用from_entries，我们将这些对组合成包含这些键和值的对象。

---

---

What (shell-assisted)

什么（shell 辅助）

This will work with a significantly older jqthan the above, and is an easily adopted approach for scenarios where a native-jqsolution can be harder to wrangle:

这将适用于jq比上述更旧的方法，并且对于本地jq解决方案难以解决的场景来说，这是一种易于采用的方法：

{
   IFS='|' read -r -a keys # read first line into an array of strings

   ## read each subsequent line into an array named "values"
   while IFS='|' read -r -a values; do

    # setup: positional arguments to pass in literal variables, query with code    
    jq_args=( )
    jq_query='.'

    # copy values into the arguments, reference them from the generated code    
    for idx in "${!values[@]}"; do
        [[ ${keys[$idx]} ]] || continue # skip values with no corresponding key
        jq_args+=( --arg "key$idx"   "${keys[$idx]}"   )
        jq_args+=( --arg "value$idx" "${values[$idx]}" )
        jq_query+=" | .[$key${idx}]=$value${idx}"
    done

    # run the generated command
    jq "${jq_args[@]}" "$jq_query" <<<'{}'
  done
} <<<"$s"

---

---

How (shell-assisted)

如何（外壳辅助）

The invoked jqcommand from the above is similar to:

jq上面调用的命令类似于：

jq --arg key0   'CONTAINER' \
   --arg value0 'nginx_container' \
   --arg key1   'CPU%' \
   --arg value1 '0.0.2%' \
   --arg key2   'MEMUSAGE/LIMIT' \
   --arg value2 '25.09MiB/15.26GiB' \
   '. | .[$key0]=$value0 | .[$key1]=$value1 | .[$key2]=$value2' \
   <<<'{}'

...passing each key and value out-of-band (such that it's treated as a literal string rather than parsed as JSON), then referring to them individually.

...在带外传递每个键和值（例如将其视为文字字符串而不是解析为 JSON），然后单独引用它们。

---

---

Result

结果

Either of the above will emit:

以上任何一个都会发出：

{
  "CONTAINER": "nginx_container",
  "CPU%": "0.02%",
  "MEMUSAGE/LIMIT": "25.09MiB/15.26GiB",
  "MEM%": "0.16%",
  "NETI/O": "0B/0B",
  "BLOCKI/O": "22.09MB/4.096kB",
  "PIDS": "0"
}

---

---

Why

为什么

In short: Because it's guaranteed to generate valid JSON as output.

简而言之：因为它保证生成有效的 JSON 作为输出。

Consider the following as an example that would break more naive approaches:

考虑以下示例，它将打破更幼稚的方法：

s='key ending in a backslash\
value "with quotes"'

Sure, these are unexpected scenarios, but jqknows how to deal with them:

当然，这些都是意想不到的情况，但jq知道如何处理它们：

{
  "key ending in a backslash\": "value \"with quotes\""
}

...whereas an implementation that didn't understand JSON strings could easily end up emitting:

...而一个不理解 JSON 字符串的实现很容易最终发出：

{
  "key ending in a backslash\": "value "with quotes""
}

回答by Jim

I know this is an old post, but the tool you seek is called jo: https://github.com/jpmens/jo

我知道这是一篇旧帖子，但您寻求的工具称为jo：https: //github.com/jpmens/jo

A quick and easy example:

一个快速简单的例子：

$ jo my_variable="simple"
{"my_variable":"simple"}

A little more complex

稍微复杂一点

$ jo -p name=jo n=17 parser=false
{
  "name": "jo",
  "n": 17,
  "parser": false
}

Add an array

添加数组

$ jo -p name=jo n=17 parser=false my_array=$(jo -a {1..5})
{
  "name": "jo",
  "n": 17,
  "parser": false,
  "my_array": [
    1,
    2,
    3,
    4,
    5
  ]
}

I've made some pretty complex stuff with jo and the nice thing is that you don't have to worry about rolling your own solution worrying about the possiblity of making invalid json.

我已经用 jo 制作了一些非常复杂的东西，好处是你不必担心滚动你自己的解决方案，担心制作无效 json 的可能性。

回答by MatrixManAtYrService

You can ask docker to give you JSON data in the first place

你可以先让docker给你JSON数据

docker stats --format "{{json .}}"

For more on this, see: https://docs.docker.com/config/formatting/

有关更多信息，请参阅：https: //docs.docker.com/config/formatting/

回答by Nick Bull

JSONSTR=""
declare -a JSONNAMES=()
declare -A JSONARRAY=()
LOOPNUM=0

cat ~/newfile | while IFS=: read CONTAINER CPU MEMUSE MEMPC NETIO BLKIO PIDS; do
    if [[ "$LOOPNUM" = 0 ]]; then
        JSONNAMES=("$CONTAINER" "$CPU" "$MEMUSE" "$MEMPC" "$NETIO" "$BLKIO" "$PIDS")
        LOOPNUM=$(( LOOPNUM+1 ))
    else
        echo "{ \"${JSONNAMES[0]}\": \"${CONTAINER}\", \"${JSONNAMES[1]}\": \"${CPU}\", \"${JSONNAMES[2]}\": \"${MEMUSE}\", \"${JSONNAMES[3]}\": \"${MEMPC}\", \"${JSONNAMES[4]}\": \"${NETIO}\", \"${JSONNAMES[5]}\": \"${BLKIO}\", \"${JSONNAMES[6]}\": \"${PIDS}\" }"
    fi 
done

Returns:

返回：

{ "CONTAINER": "nginx_container", "CPU%": "0.02%", "MEMUSAGE/LIMIT": "25.09MiB/15.26GiB", "MEM%": "0.16%", "NETI/O": "0B/0B", "BLOCKI/O": "22.09MB/4.096kB", "PIDS": "0" }

回答by jq170727

Here is a solution which uses the -Rand -soptions along with transpose:

这是一个使用-R和-s选项以及的解决方案transpose：

   split("\n")                       # [ "CONTAINER...", "nginx_container|0.02%...", ...]
 | (.[0]    | split("|")) as $keys   # [ "CONTAINER", "CPU%", "MEMUSAGE/LIMIT", ... ]
 | (.[1:][] | split("|"))            # [ "nginx_container", "0.02%", ... ] [ ... ] ...
 | select(length > 0)                # (remove empty [] caused by trailing newline)
 | [$keys, .]                        # [ ["CONTAINER", ...], ["nginx_container", ...] ] ...
 | [ transpose[] | {(.[0]):.[1]} ]   # [ {"CONTAINER": "nginx_container"}, ... ] ...
 | add                               # {"CONTAINER": "nginx_container", "CPU%": "0.02%" ...

回答by NoamG

json_template='{"CONTAINER":"%s","CPU%":"%s","MEMUSAGE/LIMIT":"%s", "MEM%":"%s","NETI/O":"%s","BLOCKI/O":"%s","PIDS":"%s"}' json_string=$(printf "$json_template" "nginx_container" "0.02%" "25.09MiB/15.26GiB" "0.16%" "0B/0B" "22.09MB/4.096kB" "0") echo "$json_string"

json_template='{"CONTAINER":"%s","CPU%":"%s","MEMUSAGE/LIMIT":"%s", "MEM%":"%s","NETI/O":"%s","BLOCKI/O":"%s","PIDS":"%s"}' json_string=$(printf "$json_template" "nginx_container" "0.02%" "25.09MiB/15.26GiB" "0.16%" "0B/0B" "22.09MB/4.096kB" "0") echo "$json_string"

Not using jq but possible to use args and environment in values.

不使用 jq，但可以在值中使用 args 和 environment。

CONTAINER=nginx_container json_template='{"CONTAINER":"%s","CPU%":"%s","MEMUSAGE/LIMIT":"%s", "MEM%":"%s","NETI/O":"%s","BLOCKI/O":"%s","PIDS":"%s"}' json_string=$(printf "$json_template" "$CONTAINER" "$1" "25.09MiB/15.26GiB" "0.16%" "0B/0B" "22.09MB/4.096kB" "0") echo "$json_string"

CONTAINER=nginx_container json_template='{"CONTAINER":"%s","CPU%":"%s","MEMUSAGE/LIMIT":"%s", "MEM%":"%s","NETI/O":"%s","BLOCKI/O":"%s","PIDS":"%s"}' json_string=$(printf "$json_template" "$CONTAINER" "$1" "25.09MiB/15.26GiB" "0.16%" "0B/0B" "22.09MB/4.096kB" "0") echo "$json_string"

回答by MatrixManAtYrService

If you're starting with tabular data, I think it makes more sense to use something that works with tabular data natively, like sqawkto make it into json, and then use jq work with it further.

如果您从表格数据开始，我认为使用本机处理表格数据的东西更有意义，例如将sqawk转换为 json，然后使用 jq 进一步处理它。

echo 'CONTAINER|CPU%|MEMUSAGE/LIMIT|MEM%|NETI/O|BLOCKI/O|PIDS
nginx_container|0.02%|25.09MiB/15.26GiB|0.16%|0B/0B|22.09MB/4.096kB|0' \
        | sqawk -FS '[|]' -RS '\n' -output json 'select * from a' header=1 \
        | jq '.[] | with_entries(select(.key|test("^a.*")|not))'

    {
      "CONTAINER": "nginx_container",
      "CPU%": "0.02%",
      "MEMUSAGE/LIMIT": "25.09MiB/15.26GiB",
      "MEM%": "0.16%",
      "NETI/O": "0B/0B",
      "BLOCKI/O": "22.09MB/4.096kB",
      "PIDS": "0"
    }

Without jq, sqawkgives a bit too much:

没有jq,sqawk有点太多了：

[
  {
    "anr": "1",
    "anf": "7",
    "a0": "nginx_container|0.02%|25.09MiB/15.26GiB|0.16%|0B/0B|22.09MB/4.096kB|0",
    "CONTAINER": "nginx_container",
    "CPU%": "0.02%",
    "MEMUSAGE/LIMIT": "25.09MiB/15.26GiB",
    "MEM%": "0.16%",
    "NETI/O": "0B/0B",
    "BLOCKI/O": "22.09MB/4.096kB",
    "PIDS": "0",
    "a8": "",
    "a9": "",
    "a10": ""
  }
]