Python 确定对象不能被腌制的原因
声明：本页面是StackOverFlow热门问题的中英对照翻译，遵循CC BY-SA 4.0协议，如果您需要使用它，必须同样遵循CC BY-SA许可，注明原文地址和作者信息，同时你必须将它归于原作者（不是我）：StackOverFlow
原文地址: http://stackoverflow.com/questions/30499341/
Warning: these are provided under cc-by-sa 4.0 license. You are free to use/share it, But you must attribute it to the original authors (not me):
StackOverFlow
Establishing why an object can't be pickled
提问by Baz
I'm receiving an object, t, from an api of type Object. I am unable to pickle it, getting the error:
我正在t从类型为 的 api接收一个对象Object。我无法腌制它，出现错误：
File "p.py", line 55, in <module>
pickle.dump(t, open('data.pkl', 'wb'))
File "/usr/lib/python2.6/pickle.py", line 1362, in dump
Pickler(file, protocol).dump(obj)
File "/usr/lib/python2.6/pickle.py", line 224, in dump
self.save(obj)
File "/usr/lib/python2.6/pickle.py", line 313, in save
(t.__name__, obj))
pickle.PicklingError: Can't pickle 'Object' object: <Object object at 0xb77b11a0>
When I do the following:
当我执行以下操作时：
for i in dir(t): print(type(i))
I get only string objects:
我只得到字符串对象：
<type 'str'>
<type 'str'>
<type 'str'>
...
<type 'str'>
<type 'str'>
<type 'str'>
How can I print the contents of my Objectobject in order to understand why it cant be pickled?
如何打印Object对象的内容以了解为什么它不能被腌制？
Its also possible that the object contains C pointers to QT objects, in which case it wouldn't make sense for me to pickle the object. But again I would like to see the internal structure of the object in order to establish this.
对象也可能包含指向 QT 对象的 C 指针，在这种情况下，我对对象进行腌制是没有意义的。但我想再次查看对象的内部结构以建立这一点。
采纳答案by bruno desthuilliers
You may want to read the python docsand check your API's Objectclass afterwards.
您可能想阅读python 文档并在Object之后检查您的 API 类。
With respect to the "internal structure of the object", usually instance attributes are stored in the __dict__attribute (and since class attributes are not pickled you only care about the instance attributes) - but note that you'll also have to recursively inspect the __dict__s for each attribute.
关于“对象的内部结构”，通常实例属性存储在__dict__属性中（并且由于类属性没有被腌制，您只关心实例属性） - 但请注意，您还必须递归检查__dict__s对于每个属性。
回答by Mike McKerns
I would use dill, which has tools to investigate what inside an object causes your target object to not be picklable. See this answer for an example: Good example of BadItem in Dill Module, and this Q&A for an example of the detection tools in real use: pandas.algos._return_false causes PicklingError with dill.dump_session on CentOS.
我会使用dill，它有工具来调查对象内部是什么导致你的目标对象不可pickle。请参阅此答案以获取示例：Dill 模块中 BadItem 的好示例，以及此问答以获取实际使用中的检测工具示例：pandas.algos._return_false 导致在 CentOS 上使用dill.dump_session 出现 PicklingError。
>>> import dill
>>> x = iter([1,2,3,4])
>>> d = {'x':x}
>>> # we check for unpicklable items in d (i.e. the iterator x)
>>> dill.detect.baditems(d)
[<listiterator object at 0x10b0e48d0>]
>>> # note that nothing inside of the iterator is unpicklable!
>>> dill.detect.baditems(x)
[]
However, the most common starting point is to use trace:
但是，最常见的起点是使用trace：
>>> dill.detect.trace(True)
>>> dill.detect.errors(d)
D2: <dict object at 0x10b8394b0>
T4: <type 'listiterator'>
PicklingError("Can't pickle <type 'listiterator'>: it's not found as __builtin__.listiterator",)
>>>
dillalso has functionality to trace pointers referrers and referents to objects, so you can build a hierarchy of how objects refer to each other. See: https://github.com/uqfoundation/dill/issues/58
dill还具有跟踪指向对象的指针引用和引用对象的功能，因此您可以构建对象如何相互引用的层次结构。见：https: //github.com/uqfoundation/dill/issues/58
Alternately, there's also: cloudpickle.py and debugpickle.py, which are for the most part no longer developed. I'm the dillauthor, and hope to soon merge any functionality in these codes that is missing in dill.
或者，还有：cloudpickle.py 和 debugpickle.py，它们大部分已不再开发。我是dill作者，希望尽快合并这些代码中缺少的任何功能dill。
回答by Alastair McCormack
I tried Dill but it didn't explain my issue. Instead, I used the following code from https://gist.github.com/andresriancho/15b5e226de68a0c2efd0, which happened to show a bug in my __getattribute__override:
我试过 Dill 但它没有解释我的问题。相反，我使用了https://gist.github.com/andresriancho/15b5e226de68a0c2efd0 中的以下代码，它碰巧在我的__getattribute__覆盖中显示了一个错误：
def debug_pickle(instance):
"""
:return: Which attribute from this object can't be pickled?
"""
attribute = None
for k, v in instance.__dict__.iteritems():
try:
cPickle.dumps(v)
except:
attribute = k
break
return attribute
Edit: Here's a reproduction of my code, using pickle and cPickle:
编辑：这是我的代码的复制，使用 pickle 和 cPickle：
class myDict(dict):
def __getattribute__(self, item):
# Try to get attribute from internal dict
item = item.replace("_", "$")
if item in self:
return self[item]
# Try super, which may leads to an AttribueError
return super(myDict, self).__getattribute__(item)
myd = myDict()
try:
with open('test.pickle', 'wb') as myf:
cPickle.dump(myd, myf, protocol=-1)
except:
print traceback.format_exc()
try:
with open('test.pickle', 'wb') as myf:
pickle.dump(myd, myf, protocol=-1)
except:
print traceback.format_exc()
Output:
输出：
Traceback (most recent call last):
File "/Users/myuser/Documents/workspace/AcceptanceTesting/ingest.py", line 35, in <module>
cPickle.dump(myd, myf, protocol=-1)
UnpickleableError: Cannot pickle <class '__main__.myDict'> objects
Traceback (most recent call last):
File "/Users/myuser/Documents/workspace/AcceptanceTesting/ingest.py", line 42, in <module>
pickle.dump(myd, myf, protocol=-1)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 1370, in dump
Pickler(file, protocol).dump(obj)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 224, in dump
self.save(obj)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 313, in save
(t.__name__, obj))
PicklingError: Can't pickle 'myDict' object: {}
You'll see that the reason is because attribute names are being mangled by __getattribute__
你会看到原因是因为属性名称被 __getattribute__
回答by Bernhard Barker
Here's an extension of Alastair's solution, in Python 3.
这是Alastair 解决方案的扩展，在 Python 3 中。
It:
它：
is recursive, to deal with complex objects where the problem might be many layers deep.
The output is in the form
.x[i].y.z....to allow you to see which members were called to get to the problem. Withdictit just prints[key/val type=...]instead, since either keys or values can be the problem, making it harder (but not impossible) to reference a specific key or value in thedict.accounts for more types, specifically
list,tupleanddict, which need to be handled separately, since they don't have__dict__attributes.returns all problems, rather than just the first one.
是递归的，用于处理问题可能有很多层深的复杂对象。
输出的形式
.x[i].y.z....允许您查看调用了哪些成员来解决问题。使用dict它只是打印[key/val type=...]，因为键或值都可能是问题所在，因此更难（但并非不可能）在dict.考虑了更多类型，特别
list是tuple和dict，它们需要单独处理，因为它们没有__dict__属性。返回所有问题，而不仅仅是第一个。
def get_unpicklable(instance, exception=None, string='', first_only=True):
"""
Recursively go through all attributes of instance and return a list of whatever
can't be pickled.
Set first_only to only print the first problematic element in a list, tuple or
dict (otherwise there could be lots of duplication).
"""
problems = []
if isinstance(instance, tuple) or isinstance(instance, list):
for k, v in enumerate(instance):
try:
pickle.dumps(v)
except BaseException as e:
problems.extend(get_unpicklable(v, e, string + f'[{k}]'))
if first_only:
break
elif isinstance(instance, dict):
for k in instance:
try:
pickle.dumps(k)
except BaseException as e:
problems.extend(get_unpicklable(
k, e, string + f'[key type={type(k).__name__}]'
))
if first_only:
break
for v in instance.values():
try:
pickle.dumps(v)
except BaseException as e:
problems.extend(get_unpicklable(
v, e, string + f'[val type={type(v).__name__}]'
))
if first_only:
break
else:
for k, v in instance.__dict__.items():
try:
pickle.dumps(v)
except BaseException as e:
problems.extend(get_unpicklable(v, e, string + '.' + k))
# if we get here, it means pickling instance caused an exception (string is not
# empty), yet no member was a problem (problems is empty), thus instance itself
# is the problem.
if string != '' and not problems:
problems.append(
string + f" (Type '{type(instance).__name__}' caused: {exception})"
)
return problems
