高性能序列化：Java vs Google Protocol Buffers vs ...？

High performance serialization: Java vs Google Protocol Buffers vs ...?

提问by cletus

For some caching I'm thinking of doing for an upcoming project, I've been thinking about Java serialization. Namely, should it be used?

对于我正在考虑为即将到来的项目做的一些缓存，我一直在考虑 Java 序列化。即，应该使用它吗？

Now I've previously written custom serialization and deserialization (Externalizable) for various reasons in years past. These days interoperability has become even more of an issue and I can foresee a need to interact with .Net applications so I've thought of using a platform-independant solution.

现在，由于过去几年的各种原因，我之前已经编写了自定义序列化和反序列化（Externalizable）。如今，互操作性已成为一个更大的问题，我可以预见需要与 .Net 应用程序进行交互，因此我考虑使用独立于平台的解决方案。

Has anyone had any experience with high-performance use of GPB? How does it compare in terms of speed and efficiency with Java's native serialization? Alternatively, are there any other schemes worth considering?

有没有人有高性能使用GPB的经验？它在速度和效率方面与 Java 的原生序列化相比如何？或者，还有其他值得考虑的方案吗？

采纳答案by Jon Skeet

I haven't compared Protocol Buffers with Java's native serialization in terms of speed, but for interoperability Java's native serialization is a serious no-no. It's also not going to be as efficient in terms of space as Protocol Buffers in most cases. Of course, it's somewhat more flexible in terms of what it can store, and in terms of references etc. Protocol Buffers is very good at what it's intended for, and when it fits your need it's great - but there are obvious restrictions due to interoperability (and other things).

我没有在速度方面将 Protocol Buffers 与 Java 的本机序列化进行比较，但对于互操作性而言，Java 的本机序列化是一个严重的禁忌。在大多数情况下，它在空间方面也不会像 Protocol Buffers 那样高效。当然，它在可以存储的内容和引用等方面更加灵活。Protocol Buffers 非常适合它的用途，并且当它满足您的需求时它很棒 - 但由于互操作性而存在明显的限制（和其他东西）。

I've recently posted a Protocol Buffers benchmarking framework in Java and .NET. The Java version is in the main Google project(in the benchmarks directory), the .NET version is in my C# port project. If you want to compare PB speed with Java serialization speed you could write similar classes and benchmark them. If you're interested in interop though, I really wouldn't give native Java serialization (or .NET native binary serialization) a second thought.

我最近在 Java 和 .NET 中发布了一个 Protocol Buffers 基准测试框架。Java 版本在主 Google 项目中（在benchmarks 目录中），.NET 版本在我的 C# 端口项目中。如果您想将 PB 速度与 Java 序列化速度进行比较，您可以编写类似的类并对它们进行基准测试。如果您对互操作感兴趣，我真的不会再考虑本机 Java 序列化（或 .NET 本机二进制序列化）。

There are other options for interoperable serialization besides Protocol Buffers though - Thrift, JSONand YAMLspring to mind, and there are doubtless others.

除了协议缓冲区之外，还有其他可互操作的序列化选项 - Thrift、JSON和YAML 浮现在脑海中，毫无疑问还有其他选项。

EDIT: Okay, with interop not being so important, it's worth trying to list the different qualities you want out of a serialization framework. One thing you should think about is versioning - this is another thing that PB is designed to handle well, both backwards and forwards (so new software can read old data and vice versa) - when you stick to the suggested rules, of course :)

编辑：好的，互操作不是那么重要，值得尝试列出您想要从序列化框架中获得的不同品质。您应该考虑的一件事是版本控制 - 这是 PB 旨在很好地处理的另一件事，无论是向后还是向前（因此新软件可以读取旧数据，反之亦然）-当然，当您坚持建议的规则时:)

Having tried to be cautious about the Java performance vs native serialization, I really wouldn't be surprised to find that PB was faster anyway. If you have the chance, use the server vm - my recent benchmarks showed the server VM to be over twice as fastat serializing and deserializing the sample data. I think the PB code suits the server VM's JIT very nicely :)

在尝试对 Java 性能与本机序列化保持谨慎之后，发现 PB 无论如何都更快，我真的不会感到惊讶。如果有机会，请使用服务器虚拟机——我最近的基准测试表明，服务器虚拟机在序列化和反序列化示例数据方面的速度是原来的两倍多。我认为 PB 代码非常适合服务器 VM 的 JIT :)

Just as sample performance figures, serializing and deserializing two messages (one 228 bytes, one 84750 bytes) I got these results on my laptop using the server VM:

正如示例性能数据，序列化和反序列化两条消息（一条 228 字节，一条 84750 字节），我使用服务器 VM 在我的笔记本电脑上得到了这些结果：

Benchmarking benchmarks.GoogleSize$SizeMessage1 with file google_message1.dat 
Serialize to byte string: 2581851 iterations in 30.16s; 18.613789MB/s 
Serialize to byte array: 2583547 iterations in 29.842s; 18.824497MB/s 
Serialize to memory stream: 2210320 iterations in 30.125s; 15.953759MB/s 
Deserialize from byte string: 3356517 iterations in 30.088s; 24.256632MB/s 
Deserialize from byte array: 3356517 iterations in 29.958s; 24.361889MB/s 
Deserialize from memory stream: 2618821 iterations in 29.821s; 19.094952MB/s 

Benchmarking benchmarks.GoogleSpeed$SpeedMessage1 with file google_message1.dat 
Serialize to byte string: 17068518 iterations in 29.978s; 123.802124MB/s 
Serialize to byte array: 17520066 iterations in 30.043s; 126.802376MB/s 
Serialize to memory stream: 7736665 iterations in 30.076s; 55.93307MB/s 
Deserialize from byte string: 16123669 iterations in 30.073s; 116.57947MB/s 
Deserialize from byte array: 16082453 iterations in 30.109s; 116.14243MB/s
Deserialize from memory stream: 7496968 iterations in 30.03s; 54.283176MB/s 

Benchmarking benchmarks.GoogleSize$SizeMessage2 with file google_message2.dat 
Serialize to byte string: 6266 iterations in 30.034s; 16.826494MB/s 
Serialize to byte array: 6246 iterations in 30.027s; 16.776697MB/s 
Serialize to memory stream: 6042 iterations in 29.916s; 16.288969MB/s 
Deserialize from byte string: 4675 iterations in 29.819s; 12.644595MB/s 
Deserialize from byte array: 4694 iterations in 30.093s; 12.580387MB/s 
Deserialize from memory stream: 4544 iterations in 29.579s; 12.389998MB/s 

Benchmarking benchmarks.GoogleSpeed$SpeedMessage2 with file google_message2.dat 
Serialize to byte string: 39562 iterations in 30.055s; 106.16416MB/s 
Serialize to byte array: 39715 iterations in 30.178s; 106.14035MB/s 
Serialize to memory stream: 34161 iterations in 30.032s; 91.74085MB/s 
Deserialize from byte string: 36934 iterations in 29.794s; 99.98019MB/s 
Deserialize from byte array: 37191 iterations in 29.915s; 100.26867MB/s 
Deserialize from memory stream: 36237 iterations in 29.846s; 97.92251MB/s 

The "speed" vs "size" is whether the generated code is optimised for speed or code size. (The serialized data is the same in both cases. The "size" version is provided for the case where you've got a lot of messages defined and don't want to take a lot of memory for the code.)

“速度”与“大小”是生成的代码是否针对速度或代码大小进行了优化。（序列化数据在这两种情况下是相同的。“大小”版本是为您定义了大量消息并且不想为代码占用大量内存的情况提供的。）

As you can see, for the smaller message it can be veryfast - over 500 small messages serialized or deserialized per millisecond. Even with the 87K message it's taking less than a millisecond per message.

如您所见，对于较小的消息，它可以非常快——每毫秒序列化或反序列化 500 多条小消息。即使使用 87K 消息，每条消息也只需要不到一毫秒。

回答by Peter Lawrey

What do you means by high performance? If you want milli-second serialization, I suggest you use the serialization approach which is simplest. If you want sub milli-second you are likely to need a binary format. If you want much below 10 micro-seconds you are likely to need a custom serialization.

你说的高性能是什么意思？如果你想要毫秒序列化，我建议你使用最简单的序列化方法。如果您想要亚毫秒，您可能需要二进制格式。如果您想要远低于 10 微秒，您可能需要自定义序列化。

I haven't seen many benchmarks for serialization/deserialization but few support less that 200 micro-seconds for serialization/deserialization.

我没有看到很多序列化/反序列化的基准测试，但很少有人支持少于 200 微秒的序列化/反序列化。

Platform independent formats come at a cost (in effort on your part and latency) you may have to decide whether you want performance or platform independence. However, there is no reason you cannot have both as a configuration option which you switch between as required.

平台独立的格式是有代价的（你的努力和延迟），你可能需要决定是想要性能还是平台独立。但是，您没有理由不能将两者都作为配置选项，您可以根据需要在两者之间进行切换。

回答by TofuBeer

Here is the off the wall suggestion of the day :-) (you just tweaked something in my head that I now want to try)...

这是当天的离墙建议:-)（你只是在我的脑海中调整了一些我现在想尝试的东西）......

If you can go for the whole caching solution via this it might work: Project Darkstar. It is designed as very high performance game server, specifically so that reads are fast (so good for a cache). It has Java and C APIs so I believe (thought it has been a long time since I looked at it, and I wasn't thinking of this then) that you could save objects with Java and read them back in C and vice versa.

如果您可以通过它获得整个缓存解决方案，它可能会起作用：Project Darkstar。它被设计为非常高性能的游戏服务器，特别是为了读取速度很快（非常适合缓存）。它有 Java 和 C API，所以我相信（认为我已经很久没有看过它了，当时我没有想到这个）你可以用 Java 保存对象并用 C 读回它们，反之亦然。

If nothing else it'll give you something to read up on today :-)

如果没有别的，它会给你一些东西今天阅读:-)

回答by instcode

If you are confusing between PB & native java serialization on speed and efficiency, just go for PB.

如果您对 PB 和本机 Java 序列化在速度和效率方面感到困惑，请选择 PB。

• PB was designed to achieve such factors. See http://code.google.com/apis/protocolbuffers/docs/overview.html
• PB data is very small while java serialization tends to replicate a whole object, including its signature. Why I always get my class name, field name... serialized, even though I know it inside out at receiver?
• Think about across language development. It's getting hard if one side uses Java, one side uses C++...

• PB 旨在实现这些因素。请参阅http://code.google.com/apis/protocolbuffers/docs/overview.html
• PB 数据非常小，而 java 序列化倾向于复制整个对象，包括其签名。为什么我总是得到我的类名，字段名......序列化，即使我在接收器里知道它的内在？
• 考虑跨语言发展。一边用Java，一边用C++，越来越难了……

Some developers suggest Thrift, but I would use Google PB because "I believe in google" :-).. Anyway, it's worth for a look: http://stuartsierra.com/2008/07/10/thrift-vs-protocol-buffers

一些开发人员建议使用 Thrift，但我会使用 Google PB，因为“我相信 google”:-)..无论如何，值得一看：http: //stuartsierra.com/2008/07/10/thrift-vs-protocol -缓冲区

回答by StaxMan

One more data point: this project:

还有一个数据点：这个项目：

http://code.google.com/p/thrift-protobuf-compare/

http://code.google.com/p/thrift-protobuf-compare/

gives some idea of expected performance for small objects, including Java serialization on PB.

给出了小对象的预期性能的一些想法，包括 PB 上的 Java 序列化。

Results vary a lot depending on your platform, but there are some general trends.

结果因您的平台而异，但有一些总体趋势。

回答by R.Moeller

You might also have a look at FST, a drop-in replacement for built-in JDK serialization that should be faster and have smaller output.

您可能还会查看FST，它是内置 JDK 序列化的替代品，它应该更快，输出更小。

raw estimations on the frequent benchmarking i have done in recent years:

对我近年来所做的频繁基准测试的原始估计：

100% = binary/struct based approaches (e.g. SBE, fst-structs)

100% = 基于二进制/结构的方法（例如 SBE、fst-structs）

• inconvenient
• postprocessing (build up "real" obejcts at receiver side) may eat up performance advantages and is never included in benchmarks

• 不方便
• 后处理（在接收端建立“真实”对象）可能会消耗性能优势，并且永远不会包含在基准测试中

~10%-35% protobuf & derivates

~10%-35% protobuf 和衍生物

~10%-30% fast serializers such as FST and KRYO

~10%-30% 快速序列化程序，例如 FST 和 KRYO

• convenient, deserialized objects can be used most often directly without additional manual translation code.
• can be pimped for performance (annotations, class registering)
• preserve links in object graph (no object serialized twice)
• can handle cyclic structures
• generic solution, FST is fully compatible to JDK serialization

• 方便，反序列化的对象可以直接使用，无需额外的手动翻译代码。
• 可以拉皮条以提高性能（注释、类注册）
• 保留对象图中的链接（没有对象序列化两次）
• 可以处理循环结构
• 通用方案，FST完全兼容JDK序列化

~2%-15% JDK serialization

~2%-15% JDK 序列化

~1%-15% fast JSon (e.g. Hymanson)

~1%-15% 快速 JSon（例如 Hymanson）

• cannot handle any object graph but only a small subset of java data structures
• no ref restoring

• 不能处理任何对象图，只能处理 Java 数据结构的一小部分
• 没有参考恢复

0.001-1% full graph JSon/XML (e.g. JSON.io)

0.001-1% 全图 JSon/XML（例如 JSON.io）

These numbers are meant to give a very rough order-of-magnitude impression. Note that performance depends A LOT on the data structures being serialized/benchmarked. So single simple class benchmarks are mostly useless (but popular: e.g. ignoring unicode, no collections, ..).

这些数字旨在给出一个非常粗略的数量级印象。请注意，性能很大程度上取决于被序列化/基准化的数据结构。所以单个简单的类基准测试大多没用（但很流行：例如忽略 unicode，没有集合，..）。

see also

也可以看看

http://java-is-the-new-c.blogspot.de/2014/12/a-persistent-keyvalue-server-in-40.html

http://java-is-the-new-c.blogspot.de/2014/12/a-persistent-keyvalue-server-in-40.html

http://java-is-the-new-c.blogspot.de/2013/10/still-using-externalizable-to-get.html

http://java-is-the-new-c.blogspot.de/2013/10/still-using-externalizable-to-get.html

回答by user4992332

For wire-friendly serialisation, consider using the Externalizable interface. Used cleverly, you'll have intimate knowlege to decide how to optimally marshall and unmarshall specific fields. That said, you'll need to manage the versioning of each object correctly - easy to un-marshall, but re-marshalling a V2 object when your code supports V1 will either break, lose information, or worse corrupt data in a way your apps aren't able to correctly process. If you're looking for an optimal path, beware no library will solve your problem without some compromises. Generally libraries will fit most use-cases and will come with the added benefit that they'll adapt and enhance over time without your input, if you've opted for an active open source project. And they might add performance problems, introduce bugs, and even fix bugs that haven't affected you yet!

对于连线友好的序列化，请考虑使用 Externalizable 接口。巧妙地使用，您将拥有丰富的知识来决定如何最佳地编组和解组特定字段。也就是说，您需要正确管理每个对象的版本控制 - 易于解组，但是当您的代码支持 V1 时重新编组 V2 对象会破坏、丢失信息，或者以您的应用程序的方式损坏数据无法正确处理。如果您正在寻找最佳路径，请注意没有任何库可以在不妥协的情况下解决您的问题。通常，如果您选择了一个活跃的开源项目，库将适合大多数用例，并且会带来额外的好处，即它们会随着时间的推移而适应和增强，而无需您的投入。他们可能会增加性能问题，引入错误，