Java 运行时生成的协议缓冲区对象

Protocol buffer objects generated at runtime

提问by Jan Zyka

a colleague of mine came with an idea of generating protocol buffers classes at runtime. Meaning:

我的一位同事提出了在运行时生成协议缓冲区类的想法。意义：

• There is C++ server application and Java client application communicating over TCP/IP via protocol buffers messages.
• The C++ application may have different schema in different versions and this is not necessarily backward compatible
• There is Java application communicating with this server which should support all possible server versions.

• C++ 服务器应用程序和 Java 客户端应用程序通过协议缓冲区消息通过 TCP/IP 进行通信。
• C++ 应用程序在不同版本中可能有不同的架构，这不一定向后兼容
• 有与此服务器通信的 Java 应用程序应该支持所有可能的服务器版本。

The idea is that the server sends the protocol buffer's definition as part of the initial handshake and the java application generates the class at runtime and use it for communication with the server.

这个想法是服务器发送协议缓冲区的定义作为初始握手的一部分，Java 应用程序在运行时生成类并使用它与服务器进行通信。

I wonder whether this is even vital idea and if there is possibly some utility for such use case.

我想知道这是否是至关重要的想法，以及是否可能有一些适用于此类用例的实用程序。

Thanks

谢谢

采纳答案by Kenton Varda

What you describe is actually already supported by the Protocol Buffers implementations in C++ and Java. All you have to do is transmit a FileDescriptorSet(as defined in google/protobuf/descriptor.proto) containing the FileDescriptorProtos representing each relevant .protofile, then use DynamicMessageto interpret the messages on the receiving end.

您描述的内容实际上已经被 C++ 和 Java 中的协议缓冲区实现支持。您所要做的就是传输一个FileDescriptorSet（如 中定义的google/protobuf/descriptor.proto），其中包含FileDescriptorProto代表每个相关.proto文件的s ，然后用于DynamicMessage在接收端解释消息。

To get a FileDescriptorProtoin C++, given message type Foothat is defined in that file, do:

要获得FileDescriptorProtoC++ 中的 a，给定Foo在该文件中定义的消息类型，请执行以下操作：

google::protobuf::FileDescriptorProto file;
Foo::descriptor().file()->CopyTo(&file);

Put all the FileDescriptorProtos that define the types you need, plus all the files that they import, into a FileDescriptorSetproto. Note that you can use google::protobuf::FileDescriptor(the thing returned by Foo::descriptor().file()) to iterate over dependencies rather than explicitly name each one.

将所有FileDescriptorProto定义您需要的类型的s 以及它们导入的所有文件放入FileDescriptorSetproto 中。请注意，您可以使用google::protobuf::FileDescriptor（由 返回的事物Foo::descriptor().file()）来迭代依赖项，而不是显式命名每个依赖项。

Now, send the FileDescriptorSetto the client.

现在，发送FileDescriptorSet给客户端。

On the client, use FileDescriptor.buildFrom()to convert each FileDescriptorPrototo a live Descriptors.FileDescriptor. You will have to make sure to build dependencies before dependents, since you have to provide the already-built dependencies to buildFrom()when building the dependents.

在客户端，用于FileDescriptor.buildFrom()将每个转换FileDescriptorProto为实时Descriptors.FileDescriptor. 您必须确保在依赖项之前构建依赖项，因为在构建依赖项时必须提供已经buildFrom()构建的依赖项。

From there, you can use the FileDescriptor's findMessageTypeByName()to find the Descriptorfor the specific message type you care about.

从那里，您可以使用FileDescriptor'sfindMessageTypeByName()来查找Descriptor您关心的特定消息类型的 。

Finally, you can call DynamicMessage.newBuilder(descriptor)to construct a new builder instance for the type in question. DynamicMessage.Builderimplements the Message.Builderinterface, which has fields like getField()and setField()to manipulate the fields of the message dynamically (by specifying the corresponding FieldDescriptors).

最后，您可以调用DynamicMessage.newBuilder(descriptor)为相关类型构造一个新的构建器实例。 DynamicMessage.Builder实现Message.Builder接口，该接口具有像getField()和setField()这样的字段，可以动态地操作消息的字段（通过指定相应的FieldDescriptors）。

Similarly, you can call DynamicMessage.parseFrom(descriptor,input)to parse messages received from the server.

同样，您可以调用DynamicMessage.parseFrom(descriptor,input)来解析从服务器收到的消息。

Note that one disadvantage of DynamicMessageis that it is relatively slow. Essentially, it's like an interpreted language. Generated code is faster because the compiler can optimize for the specific type, whereas DynamicMessagehas to be able to handle any type.

请注意， 的一个缺点DynamicMessage是它相对较慢。本质上，它就像一种解释型语言。生成的代码更快，因为编译器可以针对特定类型进行优化，而DynamicMessage必须能够处理任何类型。

However, there's really no way around this. Even if you ran the code generator and compiled the class at runtime, the code which actually usesthe new class would still be code that you wrote earlier, before you knew what type you were going to use. Therefore, it still has to use a reflection or reflection-like interface to access the message, and that is going to be slower than if the code were hand-written for the specific type.

然而，真的没有办法解决这个问题。即使您在运行时运行代码生成器并编译类，实际使用新类的代码仍然是您之前编写的代码，在您知道将使用什么类型之前。因此，它仍然必须使用反射或类似反射的接口来访问消息，这比为特定类型手写代码要慢。

But is it a good idea?

但这是个好主意吗？

Well, this depends. What is the client actually going to dowith this schema it receives from the server? Transmitting a schema over the wire doesn't magically make the client compatible with that version of the protocol -- the client still has to understandwhat the protocol means. If the protocol has been changed in a backwards-incompatible way, this almost certainly means that the meaningof the protocol has changed, and the client code has to be updated, schema transmission or not. The only time where you can expect the client to continue working without an update is when the client is only doing a generic operation that only depends on the message content but not the message meaning -- for example, the client could convert the message to JSON without having to know what it means. But this is relatively unusual, particularly on the client end of an application. This is exactly why Protobufs doesn't send any type information by default -- because it's usually useless, since if the receiver doesn't know the meaning, the schema is irrelevant.

嗯，这取决于。客户端实际上将如何处理它从服务器接收到的这个模式？通过网络传输模式并不会神奇地使客户端与该版本的协议兼容——客户端仍然必须了解协议的含义。如果协议已在向后兼容的方式被改变，这几乎肯定意味着该意思协议已经改变，客户端代码必须更新，模式传输与否。您可以期望客户端在没有更新的情况下继续工作的唯一时间是客户端只执行仅依赖于消息内容而不是消息含义的通用操作——例如，客户端可以将消息转换为 JSON无需知道这意味着什么。但这是相对不寻常的，尤其是在应用程序的客户端。这正是 Protobufs 在默认情况下不发送任何类型信息的原因——因为它通常是无用的，因为如果接收者不知道含义，则模式是无关紧要的。

If the issue is that the server is sending messages to the client which aren't intended to be interpreted at all, but just sent back to the server at a later time, then the client doesn't need the schema at all. Just transmit the message as bytesand don't bother parsing it. Note that a bytesfield containing an encoded message of type Foolooks exactly the same on the wire as a field whose type is actually declared as Foo. You could actually compile the client and server against slightly different versions of the .protofile, where the client sees a particular field as byteswhile the server sees it as a sub-message, in order to avoid the need for the client to be aware of the definition of that sub-message. ``

如果问题是服务器向客户端发送的消息根本不打算被解释，而是稍后发送回服务器，那么客户端根本不需要模式。只需将消息传输为，bytes而不必费心解析它。请注意，bytes包含类型的编码消息的字段在线路上Foo看起来与类型实际声明为 的字段完全相同Foo。您实际上可以针对略有不同版本的.proto文件编译客户端和服务器，其中客户端将特定字段视为，bytes而服务器将其视为子消息，以避免客户端需要知道定义那个子消息。``

回答by osuciu

For Java you may find following wrapper API ("protobuf-dynamic") easier to use than the original protobuf API:

对于 Java，您可能会发现以下包装器 API（“protobuf-dynamic”）比原始 protobuf API 更易于使用：

https://github.com/os72/protobuf-dynamic

https://github.com/os72/protobuf-dynamic

For example:

例如：

// Create dynamic schema
DynamicSchema.Builder schemaBuilder = DynamicSchema.newBuilder();
schemaBuilder.setName("PersonSchemaDynamic.proto");

MessageDefinition msgDef = MessageDefinition.newBuilder("Person") // message Person
    .addField("required", "int32", "id", 1)     // required int32 id = 1
    .addField("required", "string", "name", 2)  // required string name = 2
    .addField("optional", "string", "email", 3) // optional string email = 3
    .build();

schemaBuilder.addMessageDefinition(msgDef);
DynamicSchema schema = schemaBuilder.build();

// Create dynamic message from schema
DynamicMessage.Builder msgBuilder = schema.newMessageBuilder("Person");
Descriptor msgDesc = msgBuilder.getDescriptorForType();
DynamicMessage msg = msgBuilder
    .setField(msgDesc.findFieldByName("id"), 1)
    .setField(msgDesc.findFieldByName("name"), "Alan Turing")
    .setField(msgDesc.findFieldByName("email"), "[email protected]")
    .build();

Dynamic schemas can be useful in some applications to distribute changes without recompiling code (say in a more dynamically typed system). They can also be very useful for "dumb" applications that require no semantic understanding (say a data browser tool)

动态模式在某些应用程序中可用于分发更改而无需重新编译代码（例如在更动态类型的系统中）。它们对于不需要语义理解的“愚蠢”应用程序（例如数据浏览器工具）也非常有用