Python 将数据从 ctypes 数组获取到 numpy

Getting data from ctypes array into numpy

提问by dtlussier

I am using a Python (via ctypes) wrapped C library to run a series of computation. At different stages of the running, I want to get data into Python, and specifically numpyarrays.

我正在使用 Python (via ctypes) 包装的 C 库来运行一系列计算。在运行的不同阶段，我想将数据导入 Python，特别是numpy数组。

The wrapping I am using does two different types of return for array data (which is of particular interest to me):

我使用的包装对数组数据执行两种不同类型的返回（这对我特别感兴趣）：

• ctypesArray: When I do type(x)(where x is the ctypesarray, I get a <class 'module_name.wrapper_class_name.c_double_Array_12000'>in return. I know that this data is a copy of the internal data from the documentation and I am able to get it into a numpyarray easily:

  >>> np.ctypeslib.as_array(x)
  

• ctypesArray：当我这样做时type(x)（其中 x 是ctypes数组，我得到 a<class 'module_name.wrapper_class_name.c_double_Array_12000'>作为回报。我知道这个数据是文档中内部数据的副本，我可以numpy轻松地将它放入一个数组中：

  >>> np.ctypeslib.as_array(x)
  

This returns a 1D numpyarray of the data.

这将返回numpy数据的一维数组。

• ctypepointer to data: In this case from the library's documentation, I understand that I am getting a pointer to the data stored and used directly to the library. Whey I do type(y)(where y is the pointer) I get <class 'module_name.wrapper_class_name.LP_c_double'>. With this case I am still able to index through the data like y[0][2], but I was only able to get it into numpy via a super awkward:

  >>> np.frombuffer(np.core.multiarray.int_asbuffer(
      ctypes.addressof(y.contents), array_length*np.dtype(float).itemsize))
  

• ctype指向数据的指针：在这种情况下，从库的文档中，我了解到我正在获取指向存储并直接用于库的数据的指针。乳清我做type(y)（其中 y 是指针）我得到<class 'module_name.wrapper_class_name.LP_c_double'>. 在这种情况下，我仍然可以像 那样对数据进行索引y[0][2]，但我只能通过一个超级尴尬的方式将其放入 numpy：

  >>> np.frombuffer(np.core.multiarray.int_asbuffer(
      ctypes.addressof(y.contents), array_length*np.dtype(float).itemsize))
  

I found this in an old numpymailing list thread from Travis Oliphant, but not in the numpydocumentation. If instead of this approach I try as above I get the following:

我在Travis Oliphant的旧numpy邮件列表线程中找到了这个，但没有在numpy文档中找到。如果我尝试上述方法而不是这种方法，则会得到以下结果：

>>> np.ctypeslib.as_array(y)
...
...  BUNCH OF STACK INFORMATION
...
AttributeError: 'LP_c_double' object has no attribute '__array_interface__'

Is this np.frombufferapproach the best or only way to do this? I am open to other suggestions but must would still like to use numpyas I have a lot of other post-processing code that relies on numpyfunctionality that I want to use with this data.

这种np.frombuffer方法是最好的还是唯一的方法？我对其他建议numpy持开放态度，但必须仍然愿意使用，因为我有很多其他后处理代码，这些代码依赖于numpy我想与这些数据一起使用的功能。

采纳答案by Sven Marnach

Creating NumPy arrays from a ctypes pointer object is a problematic operation. It is unclear who actually owns the memory the pointer is pointing to. When will it be freed again? How long is it valid? Whenever possible I would try to avoid this kind of construct. It is so much easier and safer to create arrays in the Python code and pass them to the C function than to use memory allocated by a Python-unaware C function. By doing the latter, you negate to some extent the advantages of having a high-level language taking care of the memory management.

从 ctypes 指针对象创建 NumPy 数组是一个有问题的操作。目前尚不清楚谁实际拥有指针指向的内存。什么时候能再次释放？有效期是多久？只要有可能，我会尽量避免这种构造。在 Python 代码中创建数组并将它们传递给 C 函数比使用由 Python 不知道的 C 函数分配的内存要容易和安全得多。通过执行后者，您在某种程度上否定了使用高级语言处理内存管理的优势。

If you are really sure that someone takes care of the memory, you can create an object exposing the Python "buffer protocol" and then create a NumPy array using this buffer object. You gave one way of creating the buffer object in your post, via the undocumented int_asbuffer()function:

如果你真的确定有人负责内存，你可以创建一个暴露 Python“缓冲协议”的对象，然后使用这个缓冲对象创建一个 NumPy 数组。您通过未记录的int_asbuffer()函数提供了一种在帖子中创建缓冲区对象的方法：

buffer = numpy.core.multiarray.int_asbuffer(
    ctypes.addressof(y.contents), 8*array_length)

(Note that I substituted 8for np.dtype(float).itemsize. It's always 8, on any platform.) A different way to create the buffer object would be to call the PyBuffer_FromMemory()function from the Python C API via ctypes:

（请注意，我替换8了np.dtype(float).itemsize。在任何平台上它始终为 8。）创建缓冲区对象的另一种方法是PyBuffer_FromMemory()通过 ctypes 从 Python C API调用函数：

buffer_from_memory = ctypes.pythonapi.PyBuffer_FromMemory
buffer_from_memory.restype = ctypes.py_object
buffer = buffer_from_memory(y, 8*array_length)

For both these ways, you can create a NumPy array from bufferby

对于这两种方式，你可以从一个NumPy的阵列buffer由

a = numpy.frombuffer(buffer, float)

(I actually do not understand why you use .astype()instead of a second parameter to frombuffer; furthermore, I wonder why you use np.int, while you said earlier that the array contains doubles.)

（我实际上不明白为什么你使用.astype()而不是第二个参数 to frombuffer；此外，我想知道你为什么使用np.int，而你之前说过数组包含doubles。）

I'm afraid it won't get much easier than this, but it isn't that bad, don't you think? You could bury all the ugly details in a wrapper function and don't worry about it any more.

恐怕它不会比这更容易，但也没有那么糟糕，你不觉得吗？您可以将所有丑陋的细节都隐藏在一个包装函数中，而不必再担心了。

回答by seeker

Another possibility (which may require more recent versions of libraries than is available when the first answer was written -- I tested something similar with ctypes 1.1.0and numpy 1.5.0b2) is to convert from the pointer to the array.

另一种可能性（可能需要比编写第一个答案时可用的库版本更新 - 我用ctypes 1.1.0和测试了类似的东西numpy 1.5.0b2）是从指针转换为数组。

np.ctypeslib.as_array(
    (ctypes.c_double * array_length).from_address(ctypes.addressof(y.contents)))

This seems to still have the shared ownership semantics, so you probably need to make sure that you free the underlying buffer eventually.

这似乎仍然具有共享所有权语义，因此您可能需要确保最终释放底层缓冲区。

回答by wordy

Neither of these worked for me in Python 3. As a general solution for converting a ctypes pointer into a numpy ndarray in python 2 and 3 I found this worked (via getting a read-only buffer):

这些在 Python 3 中都不适合我。作为在 python 2 和 3 中将 ctypes 指针转换为 numpy ndarray 的一般解决方案，我发现这有效（通过获取只读缓冲区）：

def make_nd_array(c_pointer, shape, dtype=np.float64, order='C', own_data=True):
    arr_size = np.prod(shape[:]) * np.dtype(dtype).itemsize 
    if sys.version_info.major >= 3:
        buf_from_mem = ctypes.pythonapi.PyMemoryView_FromMemory
        buf_from_mem.restype = ctypes.py_object
        buf_from_mem.argtypes = (ctypes.c_void_p, ctypes.c_int, ctypes.c_int)
        buffer = buf_from_mem(c_pointer, arr_size, 0x100)
    else:
        buf_from_mem = ctypes.pythonapi.PyBuffer_FromMemory
        buf_from_mem.restype = ctypes.py_object
        buffer = buf_from_mem(c_pointer, arr_size)
    arr = np.ndarray(tuple(shape[:]), dtype, buffer, order=order)
    if own_data and not arr.flags.owndata:
        return arr.copy()
    else:
        return arr

回答by Ilya Prokin

If you are ok with creating arrays in python, the following example with 2d array works in python3:

如果您可以在 python 中创建数组，则以下带有二维数组的示例适用于 python3：

import numpy as np
import ctypes

OutType = (ctypes.c_float * 4) * 6
out = OutType()
YourCfunction = ctypes.CDLL('./yourlib.so').voidreturningfunctionwithweirdname
YourCfunction.argtypes = [ctypes.POINTER(ctypes.c_float)]*3, ctypes.POINTER(ctypes.c_float)]*5, OutType]
YourCfunction(input1, input2, out)
out = np.array(out) # convert it to numpy

print(out)

numpy and ctypes versions are 1.11.1 and 1.1.0

numpy 和 ctypes 版本是 1.11.1 和 1.1.0

回答by Eric

np.ctypeslib.as_arrayis all you need here.

np.ctypeslib.as_array这就是您所需要的一切。

From an array:

从数组：

 c_arr = (c_float * 8)()
 np.ctypeslib.as_array(c_arr)

From a pointer

从指针

 c_arr = (c_float * 8)()
 ptr = ctypes.pointer(c_arr[0])
 np.ctypeslib.as_array(ptr, shape=(8,))