oracle 根据字节长度缩短 UTF8 字符串的最佳方法

Best way to shorten UTF8 string based on byte length

提问by Michael La Voie

A recent project called for importing data into an Oracle database. The program that will do this is a C# .Net 3.5 app and I'm using the Oracle.DataAccess connection library to handle the actual inserting.

最近的一个项目要求将数据导入 Oracle 数据库。执行此操作的程序是一个 C# .Net 3.5 应用程序，我使用 Oracle.DataAccess 连接库来处理实际插入。

I ran into a problem where I'd receive this error message when inserting a particular field:

我遇到了一个问题，在插入特定字段时会收到此错误消息：

ORA-12899 Value too large for column X

ORA-12899 列 X 的值太大

I used Field.Substring(0, MaxLength);but still got the error (though not for every record).

我使用过Field.Substring(0, MaxLength);但仍然出现错误（虽然不是每条记录）。

Finally I saw what should have been obvious, my string was in ANSI and the field was UTF8. Its length is defined in bytes, not characters.

最后我看到了应该很明显的东西，我的字符串是 ANSI，字段是 UTF8。它的长度以字节为单位定义，而不是字符。

This gets me to my question. What is the best way to trim my string to fix the MaxLength?

这让我想到了我的问题。修剪字符串以修复 MaxLength 的最佳方法是什么？

My substring code works by character length. Is there simple C# function that can trim a UT8 string intelligently by byte length (ie not hack off half a character) ?

我的子字符串代码按字符长度工作。是否有简单的 C# 函数可以通过字节长度智能地修剪 UT8 字符串（即不砍掉半个字符）？

回答by Daniel Brückner

Here are two possible solution - a LINQ one-liner processing the input left to right and a traditional for-loop processing the input from right to left. Which processing direction is faster depends on the string length, the allowed byte length, and the number and distribution of multibyte characters and is hard to give a general suggestion. The decision between LINQ and traditional code I probably a matter of taste (or maybe speed).

这里有两种可能的解决方案 - 一个 LINQfor单线性处理从左到右的输入和一个传统的循环处理从右到左的输入。哪个处理方向更快取决于字符串长度、允许的字节长度以及多字节字符的数量和分布，很难给出一般性建议。LINQ 和传统代码之间的决定我可能是一个品味问题（或者可能是速度）。

If speed matters, one could think about just accumulating the byte length of each character until reaching the maximum length instead of calculating the byte length of the whole string in each iteration. But I am not sure if this will work because I don't know UTF-8 encoding well enough. I could theoreticaly imagine that the byte length of a string does not equal the sum of the byte lengths of all characters.

如果速度很重要，可以考虑只累积每个字符的字节长度直到达到最大长度，而不是在每次迭代中计算整个字符串的字节长度。但我不确定这是否可行，因为我不太了解 UTF-8 编码。我可以从理论上想象一个字符串的字节长度不等于所有字符的字节长度之和。

public static String LimitByteLength(String input, Int32 maxLength)
{
    return new String(input
        .TakeWhile((c, i) =>
            Encoding.UTF8.GetByteCount(input.Substring(0, i + 1)) <= maxLength)
        .ToArray());
}

public static String LimitByteLength2(String input, Int32 maxLength)
{
    for (Int32 i = input.Length - 1; i >= 0; i--)
    {
        if (Encoding.UTF8.GetByteCount(input.Substring(0, i + 1)) <= maxLength)
        {
            return input.Substring(0, i + 1);
        }
    }

    return String.Empty;
}

回答by ruffin

I think we can do better than naively counting the total length of a string with each addition. LINQ is cool, but it can accidentally encourage inefficient code. What if I wanted the first 80,000 bytes of a giant UTF string? That's a lotof unnecessary counting. "I've got 1 byte. Now I've got 2. Now I've got 13... Now I have 52,384..."

我认为我们可以做得比每次添加时天真地计算字符串的总长度更好。LINQ 很酷，但它可能会意外地鼓励低效代码。如果我想要一个巨大的 UTF 字符串的前 80,000 个字节怎么办？这是很多不必要的计数。“我有 1 个字节。现在我有 2 个。现在我有 13 个……现在我有 52,384 个……”

That's silly. Most of the time, at least in l'anglais, we can cut exactlyon that nthbyte. Even in another language, we're less than 6 bytes away from a good cutting point.

那是愚蠢的。大多数时候，至少在 l'anglais 中，我们可以准确地切割那个nth字节。即使在另一种语言中，我们距离一个好的切割点也只有不到 6 个字节。

So I'm going to start from @Oren's suggestion, which is to key off of the leading bit of a UTF8 char value. Let's start by cutting right at the n+1thbyte, and use Oren's trick to figure out if we need to cut a few bytes earlier.

所以我将从@Oren 的建议开始，即关闭 UTF8 字符值的前导位。让我们从直接切割n+1th字节开始，然后使用 Oren 的技巧来确定是否需要提前切割几个字节。

Three possibilities

三种可能

If the first byte after the cut has a 0in the leading bit, I know I'm cutting precisely before a single byte (conventional ASCII) character, and can cut cleanly.

如果剪切后的第一个字节在0前导位中有一个，我知道我正在精确地在单个字节（常规 ASCII）字符之前剪切，并且可以干净地剪切。

If I have a 11following the cut, the next byte after the cut is the startof a multi-byte character, so that's a good place to cut too!

如果我有一个11继切，切后的下一个字节是开始一个多字节字符的，所以这是一个好地方剪得！

If I have a 10, however, I know I'm in the middle of a multi-byte character, and need to go back to check to see where it really starts.

10但是，如果我有一个，我知道我在一个多字节字符的中间，需要回去检查它真正开始的位置。

That is, though I want to cut the string after the nth byte, if that n+1th byte comes in the middle of a multi-byte character, cutting would create an invalid UTF8 value. I need to back up until I get to one that starts with 11and cut just before it.

也就是说，尽管我想在第 n 个字节之后剪切字符串，但如果第 n+1 个字节位于多字节字符的中间，则剪切将创建无效的 UTF8 值。我需要备份，直到我找到一个开始11并在它之前剪掉的东西。

Code

代码

Notes: I'm using stuff like Convert.ToByte("11000000", 2)so that it's easy to tell what bits I'm masking (a little more about bit masking here). In a nutshell, I'm &ing to return what's in the byte's first two bits and bringing back 0s for the rest. Then I check the XXfrom XX000000to see if it's 10or 11, where appropriate.

注意：我正在使用类似的东西，Convert.ToByte("11000000", 2)以便很容易分辨出我正在屏蔽哪些位（更多关于位屏蔽在这里）。简而言之，我&将返回字节前两位中的内容，并0为其余部分带回s。然后我检查XXfromXX000000以查看它是否为10或11，在适当的情况下。

I found out todaythat C# 6.0 might actually support binary representations, which is cool, but we'll keep using this kludge for now to illustrate what's going on.

我今天发现C# 6.0 实际上可能支持二进制表示，这很酷，但我们现在将继续使用这个组合来说明发生了什么。

The PadLeftis just because I'm overly OCD about output to the Console.

这PadLeft只是因为我对控制台的输出过于强迫症。

So here's a function that'll cut you down to a string that's nbytes long or the greatest number less than nthat's ends with a "complete" UTF8 character.

因此，这里有一个函数可以将您缩减为一个n字节长的字符串或小于该字符串的最大数字n，并以“完整”的 UTF8 字符结尾。

public static string CutToUTF8Length(string str, int byteLength)
{
    byte[] byteArray = Encoding.UTF8.GetBytes(str);
    string returnValue = string.Empty;

    if (byteArray.Length > byteLength)
    {
        int bytePointer = byteLength;

        // Check high bit to see if we're [potentially] in the middle of a multi-byte char
        if (bytePointer >= 0 
            && (byteArray[bytePointer] & Convert.ToByte("10000000", 2)) > 0)
        {
            // If so, keep walking back until we have a byte starting with `11`,
            // which means the first byte of a multi-byte UTF8 character.
            while (bytePointer >= 0 
                && Convert.ToByte("11000000", 2) != (byteArray[bytePointer] & Convert.ToByte("11000000", 2)))
            {
                bytePointer--;
            }
        }

        // See if we had 1s in the high bit all the way back. If so, we're toast. Return empty string.
        if (0 != bytePointer)
        {
            returnValue = Encoding.UTF8.GetString(byteArray, 0, bytePointer); // hat tip to @NealEhardt! Well played. ;^)
        }
    }
    else
    {
        returnValue = str;
    }

    return returnValue;
}

I initially wrote this as a string extension. Just add back the thisbefore string strto put it back into extension format, of course. I removed the thisso that we could just slap the method into Program.csin a simple console app to demonstrate.

我最初把它写成一个字符串扩展。当然，只需重新添加this之前的内容string str即可将其恢复为扩展格式。我删除了 ，this以便我们可以将方法放入Program.cs一个简单的控制台应用程序中进行演示。

Test and expected output

测试和预期输出

Here's a good test case, with the output it create below, written expecting to be the Mainmethod in a simple console app's Program.cs.

这是一个很好的测试用例，它在下面创建了输出，编写的期望是Main一个简单的控制台应用程序的Program.cs.

static void Main(string[] args)
{
    string testValue = "12345“”67890”";

    for (int i = 0; i < 15; i++)
    {
        string cutValue = Program.CutToUTF8Length(testValue, i);
        Console.WriteLine(i.ToString().PadLeft(2) +
            ": " + Encoding.UTF8.GetByteCount(cutValue).ToString().PadLeft(2) +
            ":: " + cutValue);
    }

    Console.WriteLine();
    Console.WriteLine();

    foreach (byte b in Encoding.UTF8.GetBytes(testValue))
    {
        Console.WriteLine(b.ToString().PadLeft(3) + " " + (char)b);
    }

    Console.WriteLine("Return to end.");
    Console.ReadLine();
}

Output follows. Notice that the "smart quotes" in testValueare three bytes long in UTF8 (though when we write the chars to the console in ASCII, it outputs dumb quotes). Also note the ?s output for the second and third bytes of each smart quote in the output.

输出如下。请注意，testValueUTF8中的“智能引号”是三个字节长（尽管当我们以 ASCII 格式将字符写入控制台时，它会输出哑引号）。还要注意?输出中每个智能引号的第二个和第三个字节的s 输出。

The first five characters of our testValueare single bytes in UTF8, so 0-5 byte values should be 0-5 characters. Then we have a three-byte smart quote, which can't be included in its entirety until 5 + 3 bytes. Sure enough, we see that pop out at the call for 8.Our next smart quote pops out at 8 + 3 = 11, and then we're back to single byte characters through 14.

我们的前五个字符testValue是UTF8中的单字节，所以0-5字节值应该是0-5个字符。然后我们有一个三字节的智能引用，直到 5 + 3 个字节才能完整地包含它。果然，我们看到在调用8. 时弹出。我们的下一个智能引号在 8 + 3 = 11 处弹出，然后我们回到单字节字符到 14。

 0:  0::
 1:  1:: 1
 2:  2:: 12
 3:  3:: 123
 4:  4:: 1234
 5:  5:: 12345
 6:  5:: 12345
 7:  5:: 12345
 8:  8:: 12345"
 9:  8:: 12345"
10:  8:: 12345"
11: 11:: 12345""
12: 12:: 12345""6
13: 13:: 12345""67
14: 14:: 12345""678


 49 1
 50 2
 51 3
 52 4
 53 5
226 a
128 ?
156 ?
226 a
128 ?
157 ?
 54 6
 55 7
 56 8
 57 9
 48 0
226 a
128 ?
157 ?
Return to end.

So that's kind of fun, and I'm in just before the question's five year anniversary. Though Oren's description of the bits had a small error, that's exactlythe trick you want to use. Thanks for the question; neat.

所以这很有趣，而且我就在这个问题的五周年纪念日之前。尽管 Oren 对位的描述有一个小错误，但这正是您想要使用的技巧。谢谢你的提问；整洁的。

回答by Oren Trutner

If a UTF-8 bytehas a zero-valued high order bit, it's the beginning of a character. If its high order bit is 1, it's in the 'middle' of a character. The ability to detect the beginning of a character was an explicit design goal of UTF-8.

如果一个 UTF-8字节有一个零值的高位，它就是一个字符的开始。如果其高位为 1，则它位于字符的“中间”。检测字符开头的能力是 UTF-8 的一个明确设计目标。

Check out the Description section of the wikipedia articlefor more detail.

查看维基百科文章的描述部分以获取更多详细信息。

回答by firda

Shorter version of ruffin's answer. Takes advantage of the design of UTF8:

ruffin 答案的简短版本。利用UTF8 的设计：

    public static string LimitUtf8ByteCount(this string s, int n)
    {
        // quick test (we probably won't be trimming most of the time)
        if (Encoding.UTF8.GetByteCount(s) <= n)
            return s;
        // get the bytes
        var a = Encoding.UTF8.GetBytes(s);
        // if we are in the middle of a character (highest two bits are 10)
        if (n > 0 && ( a[n]&0xC0 ) == 0x80)
        {
            // remove all bytes whose two highest bits are 10
            // and one more (start of multi-byte sequence - highest bits should be 11)
            while (--n > 0 && ( a[n]&0xC0 ) == 0x80)
                ;
        }
        // convert back to string (with the limit adjusted)
        return Encoding.UTF8.GetString(a, 0, n);
    }

回答by canton7

All of the other answers appear to miss the fact that this functionality is already built into .NET, in the Encoderclass. For bonus points, this approach will also work for other encodings.

所有其他答案似乎都忽略了这个功能已经内置到 .NET 中的事实Encoder。对于奖励积分，这种方法也适用于其他编码。

public static String LimitByteLength(string input, int maxLength)
{
    if (string.IsNullOrEmpty(input) || Encoding.UTF8.GetByteLength(input) <= maxLength)
    {
        return message;
    }

    var encoder = Encoding.UTF8.GetEncoder();
    byte[] buffer = new byte[maxLength];
    char[] messageChars = message.ToCharArray();
    encoder.Convert(
        chars: messageChars,
        charIndex: 0,
        charCount: messageChars.Length,
        bytes: buffer,
        byteIndex: 0,
        byteCount: buffer.Length,
        flush: false,
        charsUsed: out int charsUsed,
        bytesUsed: out int bytesUsed,
        completed: out bool completed);

    // I don't think we can return message.Substring(0, charsUsed)
    // as that's the number of UTF-16 chars, not the number of codepoints
    // (think about surrogate pairs). Therefore I think we need to
    // actually convert bytes back into a new string
    return Encoding.UTF8.GetString(bytes, 0, bytesUsed);
}

回答by Justin Cave

Is there a reason that you need the database column to be declared in terms of bytes? That's the default, but it's not a particularly useful default if the database character set is variable width. I'd strongly prefer declaring the column in terms of characters.

是否有理由需要根据字节声明数据库列？这是默认值，但如果数据库字符集是可变宽度，则它不是特别有用的默认值。我非常喜欢用字符来声明列。

CREATE TABLE length_example (
  col1 VARCHAR2( 10 BYTE ),
  col2 VARCHAR2( 10 CHAR )
);

This will create a table where COL1 will store 10 bytes of data and col2 will store 10 characters worth of data. Character length semantics make far more sense in a UTF8 database.

这将创建一个表，其中 COL1 将存储 10 个字节的数据，而 col2 将存储 10 个字符的数据。字符长度语义在 UTF8 数据库中更有意义。

Assuming you want all the tables you create to use character length semantics by default, you can set the initialization parameter NLS_LENGTH_SEMANTICSto CHAR. At that point, any tables you create will default to using character length semantics rather than byte length semantics if you don't specify CHAR or BYTE in the field length.

假设您希望您创建的所有表默认使用字符长度语义，您可以将初始化参数设置NLS_LENGTH_SEMANTICS为 CHAR。此时，如果您未在字段长度中指定 CHAR 或 BYTE，则您创建的任何表都将默认使用字符长度语义而不是字节长度语义。

回答by Avi Pinto

Following Oren Trutner's commenthere are two more solutions to the problem:
here we count the number of bytes to remove from the end of the string according to each character at the end of the string, so we don't evaluate the entire string in every iteration.

遵循Oren Trutner 的评论，这里还有两个解决问题的方法：
这里我们根据字符串末尾的每个字符计算要从字符串末尾删除的字节数，因此我们不会在每个字符串中评估整个字符串迭代。

string str = "朣楢琴执执 瑩浻牡楧硰执执獧浻牡楧敬瑦 瀰 絸朣杢执獧扻捡杫潲湵 潣" 
int maxBytesLength = 30;
var bytesArr = Encoding.UTF8.GetBytes(str);
int bytesToRemove = 0;
int lastIndexInString = str.Length -1;
while(bytesArr.Length - bytesToRemove > maxBytesLength)
{
   bytesToRemove += Encoding.UTF8.GetByteCount(new char[] {str[lastIndexInString]} );
   --lastIndexInString;
}
string trimmedString = Encoding.UTF8.GetString(bytesArr,0,bytesArr.Length - bytesToRemove);
//Encoding.UTF8.GetByteCount(trimmedString);//get the actual length, will be <= 朣楢琴执执 瑩浻牡楧硰执执獧浻牡楧敬瑦 瀰 絸朣杢执獧扻捡杫潲湵 潣潬昣昸昸慢正 

And an even more efficient(and maintainable) solution: get the string from the bytes array according to desired length and cut the last character because it might be corrupted

还有一个更有效（和可维护）的解决方案：根据所需长度从字节数组中获取字符串并剪切最后一个字符，因为它可能已损坏

string str = "朣楢琴执执 瑩浻牡楧硰执执獧浻牡楧敬瑦 瀰 絸朣杢执獧扻捡杫潲湵 潣" 
int maxBytesLength = 30;    
string trimmedWithDirtyLastChar = Encoding.UTF8.GetString(Encoding.UTF8.GetBytes(str),0,maxBytesLength);
string trimmedString = trimmedWithDirtyLastChar.Substring(0,trimmedWithDirtyLastChar.Length - 1);

The only downside with the second solution is that we might cut a perfectly fine last character, but we are already cutting the string, so it might fit with the requirements.
Thanks to Shhadewho thought about the second solution

第二种解决方案的唯一缺点是我们可能会剪切一个完美的最后一个字符，但我们已经在剪切字符串，因此它可能符合要求。
感谢考虑第二个解决方案的Shhade

回答by Afshin

This is another solution based on binary search:

这是另一种基于二分搜索的解决方案：

public string LimitToUTF8ByteLength(string text, int size)
{
    if (size <= 0)
    {
        return string.Empty;
    }

    int maxLength = text.Length;
    int minLength = 0;
    int length = maxLength;

    while (maxLength >= minLength)
    {
        length = (maxLength + minLength) / 2;
        int byteLength = Encoding.UTF8.GetByteCount(text.Substring(0, length));

        if (byteLength > size)
        {
            maxLength = length - 1;
        }
        else if (byteLength < size)
        {
            minLength = length + 1;
        }
        else
        {
            return text.Substring(0, length); 
        }
    }

    // Round down the result
    string result = text.Substring(0, length);
    if (size >= Encoding.UTF8.GetByteCount(result))
    {
        return result;
    }
    else
    {
        return text.Substring(0, length - 1);
    }
}

回答by Anwar

public static string LimitByteLength3(string input, Int32 maxLenth)
    {
        string result = input;

        int byteCount = Encoding.UTF8.GetByteCount(input);
        if (byteCount > maxLenth)
        {
            var byteArray = Encoding.UTF8.GetBytes(input);
            result = Encoding.UTF8.GetString(byteArray, 0, maxLenth);
        }

        return result;
    }