如何将二进制数据附加到 node.js 中的缓冲区

How to append binary data to a buffer in node.js

提问by Gabriel Llamas

I have a buffer with some binary data:

我有一个包含一些二进制数据的缓冲区：

var b = new Buffer ([0x00, 0x01, 0x02]);

and I want to append 0x03.

我想附加0x03.

How can I append more binary data? I'm searching in the documentation but for appending data it must be a string, if not, an error occurs (TypeError: Argument must be a string):

如何附加更多二进制数据？我在文档中搜索，但要附加数据，它必须是一个字符串，如果不是，则会发生错误（TypeError: Argument must be a string）：

var b = new Buffer (256);
b.write ("hola");
console.log (b.toString ("utf8", 0, 4)); //hola
b.write (", adios", 4);
console.log (b.toString ("utf8", 0, 11)); //hola, adios

Then, the only solution I can see here is to create a new buffer for every appended binary data and copy it to the major buffer with the correct offset:

然后，我在这里看到的唯一解决方案是为每个附加的二进制数据创建一个新缓冲区，并将其复制到具有正确偏移量的主缓冲区：

var b = new Buffer (4); //4 for having a nice printed buffer, but the size will be 16KB
new Buffer ([0x00, 0x01, 0x02]).copy (b);
console.log (b); //<Buffer 00 01 02 00>
new Buffer ([0x03]).copy (b, 3);
console.log (b); //<Buffer 00 01 02 03>

But this seems a bit inefficient because I have to instantiate a new buffer for every append.

但这似乎有点低效，因为我必须为每个追加实例化一个新缓冲区。

Do you know a better way for appending binary data?

您知道附加二进制数据的更好方法吗？

EDIT

编辑

I've written a BufferedWriterthat writes bytes to a file using internal buffers. Same as BufferedReaderbut for writing.

我编写了一个BufferedWriter，它使用内部缓冲区将字节写入文件。与BufferedReader相同，但用于写入。

A quick example:

一个简单的例子：

//The BufferedWriter truncates the file because append == false
new BufferedWriter ("file")
    .on ("error", function (error){
        console.log (error);
    })

    //From the beginning of the file:
    .write ([0x00, 0x01, 0x02], 0, 3) //Writes 0x00, 0x01, 0x02
    .write (new Buffer ([0x03, 0x04]), 1, 1) //Writes 0x04
    .write (0x05) //Writes 0x05
    .close (); //Closes the writer. A flush is implicitly done.

//The BufferedWriter appends content to the end of the file because append == true
new BufferedWriter ("file", true)
    .on ("error", function (error){
        console.log (error);
    })

    //From the end of the file:
    .write (0xFF) //Writes 0xFF
    .close (); //Closes the writer. A flush is implicitly done.

//The file contains: 0x00, 0x01, 0x02, 0x04, 0x05, 0xFF

LAST UPDATE

最后更新

Use concat.

使用连接。

回答by Brad

Updated Answer for Node.js ~>0.8

Node.js 的更新答案 ~>0.8

Node is able to concatenate bufferson its own now.

Node 现在能够自己连接缓冲区。

var newBuffer = Buffer.concat([buffer1, buffer2]);

Old Answer for Node.js ~0.6

Node.js 的旧答案 ~0.6

I use a module to add a .concatfunction, among others:

我使用一个模块来添加一个.concat功能，其中包括：

https://github.com/coolaj86/node-bufferjs

https://github.com/coolaj86/node-bufferjs

I know it isn't a "pure" solution, but it works very well for my purposes.

我知道这不是一个“纯粹的”解决方案，但它对我的目的非常有效。

回答by stewe

Buffers are always of fixed size, there is no built in way to resize them dynamically, so your approach of copying it to a larger Buffer is the only way.

缓冲区的大小总是固定的，没有内置的方法可以动态调整它们的大小，因此将其复制到更大的缓冲区的方法是唯一的方法。

However, to be more efficient, you could make the Buffer larger than the original contents, so it contains some "free" space where you can add data without reallocating the Buffer. That way you don't need to create a new Buffer and copy the contents on each append operation.

但是，为了提高效率，您可以使 Buffer 比原始内容大，因此它包含一些“空闲”空间，您可以在其中添加数据而无需重新分配 Buffer。这样你就不需要创建一个新的 Buffer 并在每个附加操作中复制内容。

回答by kmcguire

This is to help anyone who comes here looking for a solution that wants a pure approach. I would recommend understanding this problem because it can happen in lots of different places not just with a JS Buffer object. By understanding why the problem exists and how to solve it you will improve your ability to solve other problems in the future since this one is so fundamental.

这是为了帮助来这里寻找想要纯方法的解决方案的任何人。我建议理解这个问题，因为它可能发生在很多不同的地方，而不仅仅是 JS Buffer 对象。通过了解问题存在的原因以及如何解决它，您将提高您将来解决其他问题的能力，因为这个问题非常重要。

For those of us that have to deal with these problems in other languages it is quite natural to devise a solution, but there are people who may not realize how to abstract away the complexities and implement a generally efficient dynamic buffer. The code below may have potential to be optimized further.

对于我们这些必须用其他语言处理这些问题的人来说，设计一个解决方案是很自然的，但有些人可能没有意识到如何抽象出复杂性并实现一个普遍有效的动态缓冲区。下面的代码可能有进一步优化的潜力。

I have left the read method unimplemented to keep the example small in size.

我没有实现 read 方法以保持示例的大小。

The reallocfunction in C (or any language dealing with intrinsic allocations) does not guarantee that the allocation will be expanded in size with out moving the existing data - although sometimes it is possible. Therefore most applications when needing to store a unknown amount of data will use a method like below and not constantly reallocate, unless the reallocation is very infrequent. This is essentially how most file systems handle writing data to a file. The file system simply allocates another node and keeps all the nodes linked together, and when you read from it the complexity is abstracted away so that the file/buffer appears to be a single contiguous buffer.

将realloc在C函数（或任何语言的交易具有内在分配）并不能保证分配的大小会与出把现有的数据进行扩展-虽然有时它是可能的。因此，当需要存储未知数量的数据时，大多数应用程序将使用如下所示的方法并且不会不断地重新分配，除非重新分配非常罕见。这基本上是大多数文件系统处理将数据写入文件的方式。文件系统简单地分配另一个节点并将所有节点链接在一起，当您从中读取时，复杂性被抽象掉，因此文件/缓冲区似乎是一个连续的缓冲区。

For those of you who wish to understand the difficulty in just simply providing a high performance dynamic buffer you only need to view the code below, and also do some research on memory heap algorithms and how the memory heap works for programs.

对于那些希望理解仅仅提供高性能动态缓冲区的困难的人，您只需要查看下面的代码，并对内存堆算法以及内存堆如何为程序工作进行一些研究。

Most languages will provide a fixed size buffer for performance reasons, and then provide another version that is dynamic in size. Some language systems opt for a third-party system where they keep the core functionality minimal (core distribution) and encourage developers to create libraries to solve additional or higher level problems. This is why you may question why a language does not provide some functionality. This small core functionality allows costs to be reduced in maintaining and enhancing the language, however you end up having to write your own implementations or depending on a third-party.

出于性能原因，大多数语言将提供固定大小的缓冲区，然后提供另一个大小动态的版本。一些语言系统选择第三方系统，它们将核心功能保持在最低限度（核心分布），并鼓励开发人员创建库来解决其他或更高级别的问题。这就是为什么您可能会质疑为什么一种语言不提供某些功能。这种小的核心功能可以降低维护和增强语言的成本，但是您最终必须编写自己的实现或依赖第三方。

var Buffer_A1 = function (chunk_size) {
    this.buffer_list = [];
    this.total_size = 0;
    this.cur_size = 0;
    this.cur_buffer = [];
    this.chunk_size = chunk_size || 4096;

    this.buffer_list.push(new Buffer(this.chunk_size));
};

Buffer_A1.prototype.writeByteArrayLimited = function (data, offset, length) {
    var can_write = length > (this.chunk_size - this.cur_size) ? (this.chunk_size - this.cur_size) : length;

    var lastbuf = this.buffer_list.length - 1;

    for (var x = 0; x < can_write; ++x) {
        this.buffer_list[lastbuf][this.cur_size + x] = data[x + offset];
    }

    this.cur_size += can_write;
    this.total_size += can_write;

    if (this.cur_size == this.chunk_size) {
        this.buffer_list.push(new Buffer(this.chunk_size));
        this.cur_size = 0;
    }

    return can_write;
};

/*
    The `data` parameter can be anything that is array like. It just must
    support indexing and a length and produce an acceptable value to be
    used with Buffer.
*/
Buffer_A1.prototype.writeByteArray = function (data, offset, length) {
    offset = offset == undefined ? 0 : offset;
    length = length == undefined ? data.length : length;

    var rem = length;
    while (rem > 0) {
        rem -= this.writeByteArrayLimited(data, length - rem, rem);
    }
};

Buffer_A1.prototype.readByteArray = function (data, offset, length) {
    /*
        If you really wanted to implement some read functionality
        then you would have to deal with unaligned reads which could
        span two buffers.
    */
};

Buffer_A1.prototype.getSingleBuffer = function () {
    var obuf = new Buffer(this.total_size);
    var cur_off = 0;
    var x;

    for (x = 0; x < this.buffer_list.length - 1; ++x) {
        this.buffer_list[x].copy(obuf, cur_off);
        cur_off += this.buffer_list[x].length;
    }

    this.buffer_list[x].copy(obuf, cur_off, 0, this.cur_size);

    return obuf;
};

回答by Havrylov Anton

insert byte to specific place.

将字节插入到特定位置。

insertToArray(arr,index,item) {
   return Buffer.concat([arr.slice(0,index),Buffer.from(item,"utf-8"),arr.slice(index)]);
}