如何从 linux bash 上的字符串计算 crc32 校验和

How to calculate crc32 checksum from a string on linux bash

提问by oxidworks

I used crc32 to calculate checkcums from sting long time ago, but I cannot remember how I did it.

很久以前我使用 crc32 从 sting 计算校验码，但我不记得我是怎么做的。

echo -n "LongString" | crc32    # no output

I found a solution [1] to calculate with Python, but is there not a direct way to calculate from string?

我找到了一个用 Python 计算的解决方案 [1]，但是没有直接的方法来计算字符串吗？

# signed
python -c 'import binascii; print binascii.crc32("LongString")'
python -c 'import zlib; print zlib.crc32("LongString")'
# unsigned
python -c 'import binascii; print binascii.crc32("LongString") % (1<<32)'
python -c 'import zlib; print zlib.crc32("LongString") % (1<<32)'

[1] How to calculate CRC32 with Python to match online results?

[1]如何用Python计算CRC32匹配在线结果？

回答by robert

I came up against this problem myself and I didn't want to go to the "hassle" of installing crc32. I came up with this, and although it's a little nasty it should work on most platforms, or most modern linux anyway ...

我自己遇到了这个问题，我不想去安装crc32. 我想出了这个，虽然它有点讨厌，但它应该适用于大多数平台，或者大多数现代 linux 无论如何......

echo -n "LongString" | gzip -c | tail -c8 | hexdump -n4 -e '"%u"'

Just to provide some technical details, gzip uses crc32in the last 8 bytes and the -coption causes it to output to standard output and tailstrips out the last 8 bytes.

只是为了提供一些技术细节，gzip在最后 8 个字节中使用 crc32，该-c选项使其输出到标准输出并tail去除最后 8 个字节。

hexdumpwas a little trickier and I had to futz about with it for a while before I came up with something satisfactory, but the format here seems to correctly parse the gzip crc32 as a single 32-bit number:

hexdump有点棘手，在我想出令人满意的东西之前，我不得不对它进行一段时间的讨论，但这里的格式似乎正确地将 gzip crc32 解析为单个 32 位数字：

• -n4takes only the relevant first 4 bytes of the gzip footer.
• '"%u"'is your standard fprintf format string that formats the bytes as a single unsigned 32-bit integer. Notice that there are double quotes nested within single quotes here.

• -n4只需要 gzip 页脚的前 4 个相关字节。
• '"%u"'是标准 fprintf 格式字符串，它将字节格式化为单个无符号 32 位整数。请注意，这里的单引号中嵌套了双引号。

If you want a hexadecimal checksum you can change the format string to '"%08x"'(or '"%08X"'for upper case hex) which will format the checksum as 8 character (0 padded) hexadecimal.

如果你想要一个十六进制校验和，你可以将格式字符串更改为'"%08x"'（或'"%08X"'大写十六进制），这会将校验和格式化为 8 个字符（0 填充）十六进制。

Like I say, not the most elegant solution, and perhaps not an approach you'd want to use in a performance-sensitive scenario but an approach that might appeal given the near universality of the commands used.

就像我说的，这不是最优雅的解决方案，也可能不是您想在对性能敏感的场景中使用的方法，而是考虑到所用命令几乎具有普遍性的一种方法。

The weak point here for cross-platform usability is probably the hexdumpconfiguration, since I have seen variations on it from platform to platform and it's a bit fiddly. I'd suggest if you're using this you should try some test values and compare with the results of an online tool.

跨平台可用性的弱点可能是hexdump配置，因为我已经看到它从平台到平台的变化，而且有点繁琐。我建议如果您正在使用它，您应该尝试一些测试值并与在线工具的结果进行比较。

EDITAs suggested by @PedroGimeno in the comments, you can pipe the output into odinstead of hexdumpfor identical results without the fiddly options. ... | od -t x4 -N 4 -A nfor hex ... | od -t d4 -N 4 -A nfor decimal.

编辑正如@PedroGimeno 在评论中所建议的那样，您可以将输出通过管道输入od而不是在hexdump没有繁琐选项的情况下获得相同的结果。... | od -t x4 -N 4 -A n十六进制... | od -t d4 -N 4 -A n为十进制。

回答by C Würtz

Or just use the process substitution:

或者只是使用进程替换：

crc32 <(echo "LongString")

回答by slim

On Ubuntu, at least, /usr/bin/crc32is a short Perl script, and you can see quite clearly from its source that all it can do is open files. It has no facility to read from stdin -- it doesn't have special handling for -as a filename, or a -cparameter or anything like that.

至少在 Ubuntu 上，它/usr/bin/crc32是一个简短的 Perl 脚本，您可以从它的源代码中很清楚地看到它所能做的就是打开文件。它没有从标准输入读取的功能——它没有特殊处理-作为文件名、-c参数或类似的东西。

So your easiest approach is to live with it, and make a temporary file.

所以你最简单的方法是忍受它，并制作一个临时文件。

tmpfile=$(mktemp)
echo -n "LongString" > "$tmpfile"
crc32 "$tmpfile"
rm -f "$tmpfile"

If you really don't want to write a file (e.g. it's more data than your filesystem can take -- unlikely if it's really a "long string", but for the sake for argument...) you could use a named pipe. To a simple non-random-access reader this is indistinguishable from a file:

如果你真的不想写一个文件（例如，它比你的文件系统可以接受的数据更多——如果它真的是一个“长字符串”，则不太可能，但为了参数......）你可以使用命名管道。对于一个简单的非随机访问读取器，这与文件无法区分：

fifo=$(mktemp -u)
mkfifo "$fifo"
echo -n "LongString" > "$fifo" &
crc32 "$fifo"
rm -f "$fifo"

Note the &to background the process which writes to fifo, because it will block until the next command reads it.

请注意&写入 的进程的后台fifo，因为它会阻塞，直到下一个命令读取它。

To be more fastidious about temporary file creation, see: https://unix.stackexchange.com/questions/181937/how-create-a-temporary-file-in-shell-script

要对临时文件创建更加挑剔，请参阅：https: //unix.stackexchange.com/questions/181937/how-create-a-temporary-file-in-shell-script

---

---

Alternatively, use what's in the script as an example from which to write your own Perl one-liner (the presence of crc32on your system indicates that Perl and the necessary module are installed), or use the Python one-liner you've already found.

或者，使用脚本中的内容作为示例，从中编写您自己的 Perl one-liner（crc32系统上的 表示已安装 Perl 和必要的模块），或使用您已经找到的 Python one-liner .

回答by Mark Adler

Your question already has most of the answer.

你的问题已经有了大部分答案。

echo -n 123456789 | python -c 'import sys;import zlib;print(zlib.crc32(sys.stdin.read())%(1<<32))'

correctly gives 3421780262

正确地给出 3421780262

I prefer hex:

我更喜欢十六进制：

echo -n 123456789 | python -c 'import sys;import zlib;print("%08x"%(zlib.crc32(sys.stdin.read())%(1<<32)))'
cbf43926

Be aware that there are several CRC-32 algorithms: http://reveng.sourceforge.net/crc-catalogue/all.htm#crc.cat-bits.32

请注意，有几种 CRC-32 算法：http: //reveng.sourceforge.net/crc-catalogue/all.htm#crc.cat-bits.32

回答by Léa Gris

Here is a pure Bash implementation:

这是一个纯 Bash 实现：

#!/usr/bin/env bash

declare -i -a CRC32_LOOKUP_TABLE

__generate_crc_lookup_table() {
  local -i -r LSB_CRC32_POLY=0xEDB88320 # The CRC32 polynomal LSB order
  local -i index byte lsb
  for index in {0..255}; do
    ((byte = 255 - index))
    for _ in {0..7}; do # 8-bit lsb shift
      ((lsb = byte & 0x01, byte = ((byte >> 1) & 0x7FFFFFFF) ^ (lsb == 0 ? LSB_CRC32_POLY : 0)))
    done
    ((CRC32_LOOKUP_TABLE[index] = byte))
  done
}
__generate_crc_lookup_table
typeset -r CRC32_LOOKUP_TABLE

crc32_string() {
  [[ ${#} -eq 1 ]] || return
  local -i i byte crc=0xFFFFFFFF index
  for ((i = 0; i < ${#1}; i++)); do
    byte=$(printf '%d' "'${1:i:1}") # Get byte value of character at i
    ((index = (crc ^ byte) & 0xFF, crc = (CRC32_LOOKUP_TABLE[index] ^ (crc >> 8)) & 0xFFFFFFFF))
  done
  echo $((crc ^ 0xFFFFFFFF))
}

printf 'The CRC32 of: %s\nis: %08x\n' "" "$(crc32_string "")"

# crc32_string "The quick brown fox jumps over the lazy dog"
# yields 414fa339

Testing:

测试：

bash ./crc32.sh "The quick brown fox jumps over the lazy dog"
The CRC32 of: The quick brown fox jumps over the lazy dog
is: 414fa339

回答by jimis

I use cksumand convert to hex using the shell builtin printf:

我使用cksum内置的 shell使用并转换为十六进制printf：

$ echo -n "LongString"  | cksum | cut -d\  -f1 | xargs echo printf '%0X\n' | sh
5751BDB2