如何在 C++ 中的 UTF-8 上正确使用 std::string？

How do I properly use std::string on UTF-8 in C++?

提问by stackunderflow

My platform is a Mac and C++11 (or above). I'm a C++ beginner and working on a personal project which processes Chinese and English. UTF-8 is the preferred encoding for this project.

我的平台是 Mac 和 C++11（或更高版本）。我是一个 C++ 初学者，正在做一个处理中文和英文的个人项目。UTF-8 是该项目的首选编码。

I read some posts on Stack Overflow, and many of them suggest using std::stringwhen dealing with UTF-8 and avoid wchar_tas there's no char8_tright now for UTF-8.

我在 Stack Overflow 上阅读了一些帖子，其中许多建议std::string在处理 UTF-8 时使用并避免使用，wchar_t因为现在没有char8_tUTF-8。

However, none of them talk about how to properly deal with functions like str[i], std::string::size(), std::string::find_first_of()or std::regexas these function usually returns unexpected results when facing UTF-8.

然而，他们没有谈论如何正确地与像函数处理str[i]，std::string::size()，std::string::find_first_of()或者std::regex因为这些功能通常面临UTF-8时，返回意外的结果。

Should I go ahead with std::stringor switch to std::wstring? If I should stay with std::string, what's the best practice for one to handle the above problems?

我应该继续std::string还是切换到std::wstring？如果我应该留在std::string，处理上述问题的最佳做法是什么？

回答by Matthieu M.

Unicode Glossary

Unicode 词汇表

Unicode is a vast and complex topic. I do not wish to wade too deep there, however a quick glossary is necessary:

Unicode 是一个庞大而复杂的话题。我不想在那里涉水太深，但是需要一个快速的词汇表：

1. Code Points: Code Points are the basic building blocks of Unicode, a code point is just an integer mapped to a meaning. The integer portion fits into 32 bits (well, 24 bits really), and the meaning can be a letter, a diacritic, a white space, a sign, a smiley, half a flag, ... and it can even be "the next portion reads right to left".
2. Grapheme Clusters: Grapheme Clusters are groups of semantically related Code Points, for example a flag in unicode is represented by associating two Code Points; each of those two, in isolation, has no meaning, but associated together in a Grapheme Cluster they represent a flag. Grapheme Clusters are also used to pair a letter with a diacritic in some scripts.

1. 代码点：代码点是 Unicode 的基本构建块，代码点只是一个映射到含义的整数。整数部分适合 32 位（嗯，实际上是 24 位），其含义可以是一个字母、一个变音符号、一个空格、一个符号、一个笑脸、半个旗帜……甚至可以是“下一部分从右到左阅读”。
2. Grapheme Clusters：Grapheme Clusters 是一组语义相关的 Code Points，例如 unicode 中的一个标志是通过关联两个 Code Points 来表示的；这两者中的每一个，孤立地，都没有意义，但在一个字素簇中关联在一起，它们代表一个标志。在某些脚本中，字素簇还用于将字母与变音符号配对。

This is the basic of Unicode. The distinction between Code Point and Grapheme Cluster can be mostly glossed over because for most modern languages each "character" is mapped to a single Code Point (there are dedicated accented forms for commonly used letter+diacritic combinations). Still, if you venture in smileys, flags, etc... then you may have to pay attention to the distinction.

这是Unicode的基础。Code Point 和 Grapheme Cluster 之间的区别大部分可以被掩盖，因为对于大多数现代语言，每个“字符”都映射到单个 Code Point（对于常用的字母 + 变音符号组合有专用的重音形式）。不过，如果您尝试使用笑脸、旗帜等……那么您可能需要注意区别。

---

---

UTF Primer

UTF 入门

Then, a serie of Unicode Code Points has to be encoded; the common encodings are UTF-8, UTF-16 and UTF-32, the latter two existing in both Little-Endian and Big-Endian forms, for a total of 5 common encodings.

然后，必须对一系列 Unicode 代码点进行编码；常见的编码有 UTF-8、UTF-16 和 UTF-32，后两者有 Little-Endian 和 Big-Endian 两种形式，共有 5 种常见编码。

In UTF-X, X is the size in bits of the Code Unit, each Code Point is represented as one or several Code Units, depending on its magnitude:

在 UTF-X 中，X 是Code Unit的大小（以位为单位），每个 Code Point 表示为一个或多个 Code Unit，具体取决于其大小：

• UTF-8: 1 to 4 Code Units,
• UTF-16: 1 or 2 Code Units,
• UTF-32: 1 Code Unit.

• UTF-8：1 到 4 个代码单元，
• UTF-16：1 或 2 个代码单元，
• UTF-32：1 个代码单元。

---

---

std::stringand std::wstring.

std::string和std::wstring。

1. Do not use std::wstringif you care about portability (wchar_tis only 16 bits on Windows); use std::u32stringinstead (aka std::basic_string<char32_t>).
2. The in-memory representation (std::stringor std::wstring) is independent of the on-disk representation (UTF-8, UTF-16 or UTF-32), so prepare yourself for having to convert at the boundary (reading and writing).
3. While a 32-bits wchar_tensures that a Code Unit represents a full Code Point, it still does not represent a complete Grapheme Cluster.

1. std::wstring如果您关心可移植性，请不要使用（wchar_t在 Windows 上只有 16 位）；使用std::u32string替代（又名std::basic_string<char32_t>）。
2. 内存中的表示（std::string或std::wstring）独立于磁盘上的表示（UTF-8、UTF-16 或 UTF-32），因此请准备好在边界（读取和写入）处进行转换。
3. 虽然 32 位wchar_t确保一个代码单元代表一个完整的代码点，但它仍然不代表一个完整的字素簇。

If you are only reading or composing strings, you should have no to little issues with std::stringor std::wstring.

如果您只是阅读或编写字符串，则使用std::string或应该没有什么问题std::wstring。

Troubles start when you start slicing and dicing, then you have to pay attention to (1) Code Point boundaries (in UTF-8 or UTF-16) and (2) Grapheme Clusters boundaries. The former can be handled easily enough on your own, the latter requires using a Unicode aware library.

当您开始切片和切块时，麻烦就开始了，那么您必须注意 (1) 代码点边界（在 UTF-8 或 UTF-16 中）和 (2) Grapheme Clusters 边界。前者可以很容易地由您自己处理，后者需要使用 Unicode 感知库。

---

---

Picking std::stringor std::u32string?

采摘std::string还是std::u32string？

If performance is a concern, it is likely that std::stringwill perform better due to its smaller memory footprint; though heavy use of Chinese may change the deal. As always, profile.

如果性能是一个问题，std::string由于其较小的内存占用，它的性能可能会更好；尽管大量使用中文可能会改变交易。一如既往，个人资料。

If Grapheme Clusters are not a problem, then std::u32stringhas the advantage of simplifying things: 1 Code Unit -> 1 Code Point means that you cannot accidentally split Code Points, and all the functions of std::basic_stringwork out of the box.

如果Grapheme Clusters不是问题，那么std::u32string有简化事情的好处：1 Code Unit -> 1 Code Point 意味着你不会不小心拆分Code Points，所有功能std::basic_string开箱即用。

If you interface with software taking std::stringor char*/char const*, then stick to std::stringto avoid back-and-forth conversions. It'll be a pain otherwise.

如果您与软件采用std::string或char*/接口char const*，则坚持使用std::string以避免来回转换。不然会很痛。

---

---

UTF-8 in std::string.

中的 UTF-8 std::string。

UTF-8 actually works quite well in std::string.

UTF-8 实际上在std::string.

Most operations work out of the box because the UTF-8 encoding is self-synchronizing and backward compatible with ASCII.

大多数操作都是开箱即用的，因为 UTF-8 编码是自同步的并且与 ASCII 向后兼容。

Due the way Code Points are encoded, looking for a Code Point cannot accidentally match the middle of another Code Point:

由于代码点的编码方式，查找代码点不会意外匹配另一个代码点的中间：

• str.find('\n')works,
• str.find("...")works for matching byte by byte¹,
• str.find_first_of("\r\n")works if searching for ASCII characters.

• str.find('\n')作品，
• str.find("...")适用于逐字节匹配¹，
• str.find_first_of("\r\n")作品如果搜索ASCII字符。

Similarly, regexshould mostly works out of the box. As a sequence of characters ("haha") is just a sequence of bytes ("哈"), basic search patterns should work out of the box.

同样，regex应该大多是开箱即用的。由于字符序列 ( "haha") 只是字节序列 ( "哈")，因此基本的搜索模式应该是开箱即用的。

Be wary, however, of character classes (such as [:alphanum:]), as depending on the regex flavor and implementation it may or may not match Unicode characters.

但是，请注意字符类（例如[:alphanum:]），因为根据正则表达式的风格和实现，它可能匹配也可能不匹配 Unicode 字符。

Similarly, be wary of applying repeaters to non-ASCII "characters", "哈?"may only consider the last byte to be optional; use parentheses to clearly delineate the repeated sequence of bytes in such cases: "(哈)?".

类似地，警惕将转发器应用于非 ASCII“字符”，"哈?"可能只考虑最后一个字节是可选的；使用括号来明确界定在这样的情况下字节重复序列："(哈)?"。

¹The key concepts to look-up are normalization and collation; this affects all comparison operations. std::stringwill always compare (and thus sort) byte by byte, without regard for comparison rules specific to a language or a usage. If you need to handle full normalization/collation, you need a complete Unicode library, such as ICU.

¹查找的关键概念是规范化和整理；这会影响所有比较操作。std::string将始终逐字节比较（并因此排序），而不考虑特定于语言或用法的比较规则。如果您需要处理完全规范化/整理，则需要一个完整的 Unicode 库，例如 ICU。

回答by James Picone

std::stringand friends are encoding-agnostic. The only difference between std::wstringand std::stringare that std::wstringuses wchar_tas the individual element, not char. For most compilers the latter is 8-bit. The former is supposed to be large enough to hold any unicode character, but in practice on some systems it isn't (Microsoft's compiler, for example, uses a 16-bit type). You can't store UTF-8 in std::wstring; that's not what it's designed for. It's designed to be an equivalent of UTF-32 - a string where each element is a single Unicode codepoint.

std::string和朋友是编码不可知的。唯一的区别std::wstring和std::string是std::wstring用途wchar_t作为单独的元素，不是char。对于大多数编译器，后者是 8 位的。前者应该足够大以容纳任何 unicode 字符，但实际上在某些系统上并非如此（例如，Microsoft 的编译器使用 16 位类型）。您不能将 UTF-8 存储在std::wstring; 这不是它的设计目的。它被设计为等效于 UTF-32 - 一个字符串，其中每个元素都是一个 Unicode 代码点。

If you want to index UTF-8 strings by Unicode codepoint or composed unicode glyph (or some other thing), count the length of a UTF-8 string in Unicode codepoints or some other unicode object, or find by Unicode codepoint, you're going to need to use something other than the standard library. ICUis one of the libraries in the field; there may be others.

如果你想通过 Unicode 代码点或组合的 unicode 字形（或其他东西）索引 UTF-8 字符串，计算 Unicode 代码点或其他一些 unicode 对象中 UTF-8 字符串的长度，或者通过 Unicode 代码点查找，你是将需要使用标准库以外的东西。ICU是该领域的图书馆之一；可能还有其他人。

Something that's probably worth noting is that if you're searching for ASCII characters, you can mostly treat a UTF-8 bytestream as if it were byte-by-byte. Each ASCII character encodes the same in UTF-8 as it does in ASCII, and every multi-byte unit in UTF-8 is guaranteed not to include any bytes in the ASCII range.

可能值得注意的一点是，如果您正在搜索 ASCII 字符，您通常可以将 UTF-8 字节流视为逐字节处理。每个 ASCII 字符在 UTF-8 中的编码方式与在 ASCII 中相同，并且 UTF-8 中的每个多字节单元都保证不包含 ASCII 范围内的任何字节。

回答by zneak

Both std::string?and std::wstringmust use UTF encoding to represent Unicode. On macOS specifically, std::stringis UTF-8 (8-bit code units), and std::wstring?is UTF-32 (32-bit code units); note that the size of wchar_tis platform-dependent.

无论std::string？并且std::wstring必须使用UTF编码来表示Unicode。特别std::string是在 macOS 上，是 UTF-8（8 位代码单元），而std::wstring? 是 UTF-32（32 位代码单元）；请注意，的大小wchar_t取决于平台。

For both, sizetracks the number of code units instead of the number of code points, or grapheme clusters. (A code point is one named Unicode entity, one or more of which form a grapheme cluster. Grapheme clusters are the visible characters that users interact with, like letters or emojis.)

对于两者，size跟踪代码单元的数量而不是代码点或字素簇的数量。（代码点是一个命名的 Unicode 实体，其中一个或多个形成一个字素簇。字素簇是用户与之交互的可见字符，如字母或表情符号。）

Although I'm not familiar with the Unicode representation of Chinese, it's very possible that when you use UTF-32, the number of code units is often very close to the number of grapheme clusters. Obviously, however, this comes at the cost of using up to 4x more memory.

虽然我对中文的Unicode表示不熟悉，但是很有可能在你使用UTF-32的时候，代码单元的数量往往和字素簇的数量非常接近。然而，显然，这是以使用多达 4 倍的内存为代价的。

The most accurate solution would be to use a Unicode library, such as ICU, to calculate the Unicode properties that you are after.

最准确的解决方案是使用 Unicode 库（例如 ICU）来计算您需要的 Unicode 属性。

Finally, UTF strings in human languages that don't use combining characters usually do pretty well with find/regex. I'm not sure about Chinese, but English is one of them.

最后，不使用组合字符的人类语言中的 UTF 字符串通常与find/配合得很好regex。我不确定中文，但英文是其中之一。

回答by Lyberta

Consider upgrading to C++20 and std::u8stringthat is the best thing we have as of 2019 for holding UTF-8. There are no standard library facilities to access individual code points or grapheme clusters but at least your type is strong enough to at least say it is true UTF-8.

考虑升级到 C++20，std::u8string这是截至 2019 年我们拥有的最好的东西来保存 UTF-8。没有标准的库工具来访问单个代码点或字素簇，但至少你的类型足够强大，至少可以说它是真正的 UTF-8。