在 Java JNI 中获取真正的 UTF-8 字符
声明：本页面是StackOverFlow热门问题的中英对照翻译，遵循CC BY-SA 4.0协议，如果您需要使用它，必须同样遵循CC BY-SA许可，注明原文地址和作者信息，同时你必须将它归于原作者（不是我）：StackOverFlow
原文地址: http://stackoverflow.com/questions/32205446/
Warning: these are provided under cc-by-sa 4.0 license. You are free to use/share it, But you must attribute it to the original authors (not me):
StackOverFlow
Getting true UTF-8 characters in Java JNI
提问by Rolf
Is there an easy way to convert a Java string to a true UTF-8 byte array in JNI code?
有没有一种简单的方法可以在 JNI 代码中将 Java 字符串转换为真正的 UTF-8 字节数组？
Unfortunately GetStringUTFChars() almostdoes what's required but not quite, it returns a "modified" UTF-8 byte sequence. The main difference is that a modified UTF-8 doesn't contain any null characters (so you can treat is an ANSI C null terminated string) but another difference seems to be how Unicode supplementary characters such as emoji are treated.
不幸的是 GetStringUTFChars()几乎完成了所需但不完全的，它返回一个“修改过的”UTF-8 字节序列。主要区别在于修改后的 UTF-8 不包含任何空字符（因此您可以处理 ANSI C 以空字符结尾的字符串），但另一个区别似乎是如何处理 Unicode 补充字符（例如表情符号）。
A character such as U+1F604 "SMILING FACE WITH OPEN MOUTH AND SMILING EYES" is stored as a surrogate pair (two UTF-16 characters U+D83D U+DE04) and has a 4-byte UTF-8 equivalent of F0 9F 98 84, and that is the byte sequence that I get if I convert the string to UTF-8 in Java:
像 U+1F604“SMILING FACE WITH OPEN MOUTH AND SMILING EYES”这样的字符被存储为一个代理对（两个 UTF-16 字符 U+D83D U+DE04），并且有一个 4 字节的 UTF-8 等效于 F0 9F 98 84，这是我在 Java 中将字符串转换为 UTF-8 时得到的字节序列：
char[] c = Character.toChars(0x1F604);
String s = new String(c);
System.out.println(s);
for (int i=0; i<c.length; ++i)
System.out.println("c["+i+"] = 0x"+Integer.toHexString(c[i]));
byte[] b = s.getBytes("UTF-8");
for (int i=0; i<b.length; ++i)
System.out.println("b["+i+"] = 0x"+Integer.toHexString(b[i] & 0xFF));
The code above prints the following:
上面的代码打印以下内容：
c[0] = 0xd83d c[1] = 0xde04 b[0] = 0xf0 b[1] = 0x9f b[2] = 0x98 b[3] = 0x84
c[0] = 0xd83d c[1] = 0xde04 b[0] = 0xf0 b[1] = 0x9f b[2] = 0x98 b[3] = 0x84
However, if I pass 's' into a native JNI method and call GetStringUTFChars() I get 6 bytes. Each of the surrogate pair characters is being converted to a 3-byte sequence independently:
但是，如果我将 's' 传递给本机 JNI 方法并调用 GetStringUTFChars() 我得到 6 个字节。每个代理对字符都被独立转换为 3 字节序列：
JNIEXPORT void JNICALL Java_EmojiTest_nativeTest(JNIEnv *env, jclass cls, jstring _s)
{
const char* sBytes = env->GetStringUTFChars(_s, NULL);
for (int i=0; sBytes[i]!=0; ++i)
fprintf(stderr, "%d: %02x\n", i, sBytes[i]);
env->ReleaseStringUTFChars(_s, sBytes);
return result;
}
0: ed 1: a0 2: bd 3: ed 4: b8 5: 84
0: ed 1: a0 2: bd 3: ed 4: b8 5: 84
The Wikipedia UTF-8 articlesuggests that GetStringUTFChars() actually returns CESU-8 rather than UTF-8. That in turn causes my native Mac code to crash because it's not a valid UTF-8 sequence:
在维基百科UTF-8的文章表明，GetStringUTFChars（）实际上返回CESU-8而不是UTF-8。这反过来会导致我的本机 Mac 代码崩溃，因为它不是有效的 UTF-8 序列：
CFStringRef str = CFStringCreateWithCString(NULL, path, kCFStringEncodingUTF8);
CFURLRef url = CFURLCreateWithFileSystemPath(NULL, str, kCFURLPOSIXPathStyle, false);
I suppose I could change all my JNI methods to take a byte[] rather than a String and do the UTF-8 conversion in Java but that seems a bit ugly, is there a better solution?
我想我可以改变我所有的 JNI 方法来获取一个 byte[] 而不是一个 String 并在 Java 中进行 UTF-8 转换，但这看起来有点难看，有没有更好的解决方案？
回答by Remy Lebeau
This is clearly explained in the Java documentation:
这在 Java 文档中有清楚的解释：
GetStringUTFChars
const char * GetStringUTFChars(JNIEnv *env, jstring string, jboolean *isCopy);Returns a pointer to an array of bytes representing the string in modified UTF-8 encoding. This array is valid until it is released by ReleaseStringUTFChars().
获取字符串UTFChars
const char * GetStringUTFChars(JNIEnv *env, jstring string, jboolean *isCopy);返回一个指向字节数组的指针，该数组表示修改后的 UTF-8 编码中的字符串。该数组在被 ReleaseStringUTFChars() 释放之前一直有效。
The JNI uses modified UTF-8 strings to represent various string types. Modified UTF-8 strings are the same as those used by the Java VM. Modified UTF-8 strings are encoded so that character sequences that contain only non-null ASCII characters can be represented using only one byte per character, but all Unicode characters can be represented.
All characters in the range
\u0001to\u007Fare represented by a single byte, as follows:
The seven bits of data in the byte give the value of the character represented.
The null character (
'\u0000') and characters in the range'\u0080'to'\u07FF'are represented by a pair of bytes x and y:
The bytes represent the character with the value
((x & 0x1f) << 6) + (y & 0x3f).Characters in the range
'\u0800'to'\uFFFF'are represented by 3 bytes x, y, and z:
The character with the value
((x & 0xf) << 12) + ((y & 0x3f) << 6) + (z & 0x3f)is represented by the bytes.Characters with code points above U+FFFF (so-called supplementary characters) are represented by separately encoding the two surrogate code units of their UTF-16 representation. Each of the surrogate code units is represented by three bytes. This means, supplementary characters are represented by six bytes, u, v, w, x, y, and z:
The character with the value
0x10000+((v&0x0f)<<16)+((w&0x3f)<<10)+(y&0x0f)<<6)+(z&0x3f)is represented by the six bytes.The bytes of multibyte characters are stored in the class file in big-endian (high byte first) order.
There are two differences between this format and the standard UTF-8 format. First, the null character (char)0 is encoded using the two-byte format rather than the one-byte format. This means that modified UTF-8 strings never have embedded nulls. Second, only the one-byte, two-byte, and three-byte formats of standard UTF-8 are used. The Java VM does not recognize the four-byte format of standard UTF-8; it uses its own two-times-three-byte format instead.
For more information regarding the standard UTF-8 format, see section 3.9 Unicode Encoding Forms of The Unicode Standard, Version 4.0.
JNI 使用修改后的 UTF-8 字符串来表示各种字符串类型。修改后的 UTF-8 字符串与 Java VM 使用的字符串相同。对修改后的 UTF-8 字符串进行编码，以便仅包含非空 ASCII 字符的字符序列可以使用每个字符仅一个字节来表示，但可以表示所有 Unicode 字符。
\u0001to范围内的所有字符\u007F都由单个字节表示，如下所示：
字节中的七位数据给出了所表示字符的值。
空字符（
'\u0000'）和字符的范围内'\u0080'，以'\u07FF'由一对字节x和y表示：
字节代表具有值的字符
((x & 0x1f) << 6) + (y & 0x3f)。
'\u0800'to范围内的字符'\uFFFF'由 3 个字节 x、y 和 z 表示：
带有值的字符
((x & 0xf) << 12) + ((y & 0x3f) << 6) + (z & 0x3f)由字节表示。代码点高于 U+FFFF 的字符（所谓的增补字符）通过分别编码其 UTF-16 表示的两个代理代码单元来表示。每个代理代码单元由三个字节表示。这意味着，补充字符由六个字节 u、v、w、x、y 和 z 表示：
具有值的字符
0x10000+((v&0x0f)<<16)+((w&0x3f)<<10)+(y&0x0f)<<6)+(z&0x3f)由六个字节表示。多字节字符的字节以大端（高字节在前）顺序存储在类文件中。
这种格式与标准的 UTF-8 格式有两个区别。首先，空字符 (char)0 使用两字节格式而不是单字节格式进行编码。这意味着修改后的 UTF-8 字符串永远不会嵌入空值。其次，仅使用标准 UTF-8 的一字节、二字节和三字节格式。Java VM 无法识别标准 UTF-8 的四字节格式；它使用自己的两倍三字节格式。
有关标准 UTF-8 格式的更多信息，请参阅 Unicode 标准 4.0 版的 3.9 Unicode 编码形式部分。
Since U+1F604 is a supplementary character, and Java does not support UTF-8's 4-byte encoding format, U+1F604 is represented in modified UTF-8 by encoding the UTF-16 surrogate pair U+D83D U+DE04using 3 bytes per surrogate, thus 6 bytes total.
由于 U+1F604 是补充字符，而 Java 不支持 UTF-8 的 4 字节编码格式，因此 U+1F604 以修改后的 UTF-8 表示，通过对 UTF-16 代理对U+D83D U+DE04进行编码，每个代理使用 3 个字节，即 6 个字节全部的。
So, to answer your question...
所以，回答你的问题...
Is there an easy way to convert a Java string to a true UTF-8 byte array in JNI code?
有没有一种简单的方法可以在 JNI 代码中将 Java 字符串转换为真正的 UTF-8 字节数组？
You can either:
您可以：
Use
GetStringChars()to get the original UTF-16 encoded characters, and then create your own UTF-8 byte array from that. The conversion from UTF-16 to UTF-8 is a very simply algorithm to implement by hand, or you can use any pre-existing implementation provided by your platform or 3rd party libraries.Have your JNI code call back into Java to invoke the
String.getBytes(String charsetName)method to encode thejstringobject to a UTF-8 byte array, eg:JNIEXPORT void JNICALL Java_EmojiTest_nativeTest(JNIEnv *env, jclass cls, jstring _s) { const jclass stringClass = env->GetObjectClass(_s); const jmethodID getBytes = env->GetMethodID(stringClass, "getBytes", "(Ljava/lang/String;)[B"); const jstring charsetName = env->NewStringUTF("UTF-8"); const jbyteArray stringJbytes = (jbyteArray) env->CallObjectMethod(_s, getBytes, charsetName); env->DeleteLocalRef(charsetName); const jsize length = env->GetArrayLength(stringJbytes); const jbyte* pBytes = env->GetByteArrayElements(stringJbytes, NULL); for (int i = 0; i < length; ++i) fprintf(stderr, "%d: %02x\n", i, pBytes[i]); env->ReleaseByteArrayElements(stringJbytes, pBytes, JNI_ABORT); env->DeleteLocalRef(stringJbytes); }
使用
GetStringChars()得到原始UTF-16编码的字符，然后创建你自己的UTF-8字节数组。从 UTF-16 到 UTF-8 的转换是一种非常简单的算法，可以手动实现，或者您可以使用平台或第三方库提供的任何预先存在的实现。让您的 JNI 代码回调到 Java 以调用
String.getBytes(String charsetName)将jstring对象编码为 UTF-8 字节数组的方法，例如：JNIEXPORT void JNICALL Java_EmojiTest_nativeTest(JNIEnv *env, jclass cls, jstring _s) { const jclass stringClass = env->GetObjectClass(_s); const jmethodID getBytes = env->GetMethodID(stringClass, "getBytes", "(Ljava/lang/String;)[B"); const jstring charsetName = env->NewStringUTF("UTF-8"); const jbyteArray stringJbytes = (jbyteArray) env->CallObjectMethod(_s, getBytes, charsetName); env->DeleteLocalRef(charsetName); const jsize length = env->GetArrayLength(stringJbytes); const jbyte* pBytes = env->GetByteArrayElements(stringJbytes, NULL); for (int i = 0; i < length; ++i) fprintf(stderr, "%d: %02x\n", i, pBytes[i]); env->ReleaseByteArrayElements(stringJbytes, pBytes, JNI_ABORT); env->DeleteLocalRef(stringJbytes); }
The Wikipedia UTF-8 article suggests that GetStringUTFChars() actually returns CESU-8 rather than UTF-8
维基百科 UTF-8 文章表明 GetStringUTFChars() 实际上返回 CESU-8 而不是 UTF-8
Java's Modified UTF-8 is not exactly the same as CESU-8:
Java 的 Modified UTF-8 与CESU-8并不完全相同：
CESU-8 is similar to Java's Modified UTF-8 but does not have the special encoding of the NUL character (U+0000).
CESU-8 类似于 Java 的 Modified UTF-8，但没有 NUL 字符 (U+0000) 的特殊编码。
