python 如何判断哪个对象属性泡菜失败？

How to tell for which object attribute pickle fails?

提问by nikow

When you pickle an object that has some attributes which cannot be pickled it will fail with a generic error message like:

当您腌制具有某些无法腌制的属性的对象时，它将失败并显示一般错误消息，例如：

PicklingError: Can't pickle <type 'instancemethod'>: attribute lookup __builtin__.instancemethod failed

Is there any way to tell which attribute caused the exception? I am using Python 2.5.2.

有什么办法可以判断是哪个属性导致了异常？我正在使用 Python 2.5.2。

Even though I understand in principle the root cause of the problem (e.g. in the above example having an instance method) it can still be very hard to exactly pinpoint it. In my case I already defined a custom __getstate__method, but forgot about a critical attribute. This happened in a complicated structure of nested objects, so it took me a while to identify the bad attribute.

即使我原则上理解问题的根本原因（例如，在上面的例子中有一个实例方法），但仍然很难准确地查明它。就我而言，我已经定义了一个自定义__getstate__方法，但忘记了一个关键属性。这发生在嵌套对象的复杂结构中，所以我花了一段时间来识别错误的属性。

As requested, here is one simple example were pickle intentionally fails:

根据要求，这是一个简单的例子，pickle 故意失败：

import cPickle as pickle
import new

class Test(object):
    pass

def test_func(self):
    pass

test = Test()
pickle.dumps(test)
print "now with instancemethod..."
test.test_meth = new.instancemethod(test_func, test)
pickle.dumps(test)

This is the output:

这是输出：

now with instancemethod...
Traceback (most recent call last):
  File "/home/wilbert/develop/workspace/Playground/src/misc/picklefail.py", line 15, in <module>
    pickle.dumps(test)
  File "/home/wilbert/lib/python2.5/copy_reg.py", line 69, in _reduce_ex
    raise TypeError, "can't pickle %s objects" % base.__name__
TypeError: can't pickle instancemethod objects

Unfortunately there is no hint that the attribute test_methcauses the problem.

不幸的是，没有迹象表明该属性test_meth会导致问题。

采纳答案by joeforker

You could file a bug against Python for not including more helpful error messages. In the meantime, modify the _reduce_ex()function in copy_reg.py.

您可以针对 Python 提交一个错误，因为它不包含更多有用的错误消息。同时，修改_reduce_ex()的功能copy_reg.py。

if base is self.__class__:
    print self # new   
    raise TypeError, "can't pickle %s objects" % base.__name__

Output:

输出：

<bound method ?.test_func of <__main__.Test object at 0xb7f4230c>>
Traceback (most recent call last):
  File "nopickle.py", line 14, in ?
    pickle.dumps(test)
  File "/usr/lib/python2.4/copy_reg.py", line 69, in _reduce_ex
    raise TypeError, "can't pickle %s objects" % base.__name__
TypeError: can't pickle instancemethod objects

回答by M.D.

I had the same problem as you, but my classes were a bit more complicated (i.e. a large tree of similar objects) so the printing didn't help much so I hacked together a helper function. It is not complete and is only intended for use with pickling protocol 2: It was enough so I could locate my problems. If you want to extend it to cover everything, the protocol is described at http://www.python.org/dev/peps/pep-0307/i've made this post editable so everybody can update the code.

我和你有同样的问题，但我的类有点复杂（即类似对象的大树）所以打印没有太大帮助，所以我把一个辅助函数组合在一起。它并不完整，仅适用于酸洗协议 2：这足以让我找到我的问题。如果您想扩展它以涵盖所有内容，该协议在http://www.python.org/dev/peps/pep-0307/ 中进行了描述，我已将此帖子设为可编辑，以便每个人都可以更新代码。

import pickle
def get_pickling_errors(obj,seen=None):
    if seen == None:
        seen = []
    try:
        state = obj.__getstate__()
    except AttributeError:
        return
    if state == None:
        return
    if isinstance(state,tuple):
        if not isinstance(state[0],dict):
            state=state[1]
        else:
            state=state[0].update(state[1])
    result = {}    
    for i in state:
        try:
            pickle.dumps(state[i],protocol=2)
        except pickle.PicklingError:
            if not state[i] in seen:
                seen.append(state[i])
                result[i]=get_pickling_errors(state[i],seen)
    return result

An example of the usage where K is the object that doesn't pickle

K 是不腌制的对象的用法示例

>>> get_pickling_errors(K)
{'_gen': {}, '_base': {'_gens': None}}

This means that the attibute K._gen is not picklable and the same goes for K._base._gens.

这意味着属性 K._gen 不可选择，K._base._gens 也是如此。

回答by Stuart Mitchell

I have found that if you subclass Pickler and wrap the Pickler.save() method in a try, except block

我发现如果你子类化 Pickler 并在 try 中包装 Pickler.save() 方法，除了块

import pickle
class MyPickler (pickle.Pickler):
    def save(self, obj):
        try:
            pickle.Pickler.save(self, obj)
        except Exception, e:
            import pdb;pdb.set_trace()

Then call it like so

然后像这样调用它

import StringIO
output = StringIO.StringIO()
MyPickler(output).dump(thingee)

回答by Mike McKerns

If you use dill, your example doesn't fail to pickle...

如果您使用dill，您的示例不会失败...

>>> import dill
>>> import new
>>> 
>>> class Test(object):
...     pass
... 
>>> def test_func(self):
...     pass
... 
>>> test = Test()
>>> dill.dumps(test)
'\x80\x02cdill.dill\n_create_type\nq\x00(cdill.dill\n_load_type\nq\x01U\x08TypeTypeq\x02\x85q\x03Rq\x04U\x04Testq\x05h\x01U\nObjectTypeq\x06\x85q\x07Rq\x08\x85q\t}q\n(U\r__slotnames__q\x0b]q\x0cU\n__module__q\rU\x08__main__q\x0eU\x07__doc__q\x0fNutq\x10Rq\x11)\x81q\x12}q\x13b.'
>>> test.test_meth = new.instancemethod(test_func, test)
>>> dill.dumps(test)
'\x80\x02cdill.dill\n_create_type\nq\x00(cdill.dill\n_load_type\nq\x01U\x08TypeTypeq\x02\x85q\x03Rq\x04U\x04Testq\x05h\x01U\nObjectTypeq\x06\x85q\x07Rq\x08\x85q\t}q\n(U\r__slotnames__q\x0b]q\x0cU\n__module__q\rU\x08__main__q\x0eU\x07__doc__q\x0fNutq\x10Rq\x11)\x81q\x12}q\x13U\ttest_methq\x14h\x01U\nMethodTypeq\x15\x85q\x16Rq\x17cdill.dill\n_create_function\nq\x18(cdill.dill\n_unmarshal\nq\x19Ubc\x01\x00\x00\x00\x01\x00\x00\x00\x01\x00\x00\x00C\x00\x00\x00s\x04\x00\x00\x00d\x00\x00S(\x01\x00\x00\x00N(\x00\x00\x00\x00(\x01\x00\x00\x00t\x04\x00\x00\x00self(\x00\x00\x00\x00(\x00\x00\x00\x00s\x07\x00\x00\x00<stdin>t\t\x00\x00\x00test_func\x01\x00\x00\x00s\x02\x00\x00\x00\x00\x01q\x1a\x85q\x1bRq\x1cc__builtin__\n__main__\nU\ttest_funcq\x1dNN}q\x1etq\x1fRq h\x12N\x87q!Rq"sb.'

So we have to find something that dillcan't pickle...

所以我们必须找到dill不能腌制的东西......

>>> class Unpicklable(object):
...   def breakme(self):
...     self.x = iter(set())
... 
>>> u = Unpicklable()
>>> dill.dumps(u)
'\x80\x02cdill.dill\n_create_type\nq\x00(cdill.dill\n_load_type\nq\x01U\x08TypeTypeq\x02\x85q\x03Rq\x04U\x0bUnpicklableq\x05h\x01U\nObjectTypeq\x06\x85q\x07Rq\x08\x85q\t}q\n(U\r__slotnames__q\x0b]q\x0cU\n__module__q\rU\x08__main__q\x0eU\x07breakmeq\x0fcdill.dill\n_create_function\nq\x10(cdill.dill\n_unmarshal\nq\x11U\xafc\x01\x00\x00\x00\x02\x00\x00\x00\x02\x00\x00\x00C\x00\x00\x00s"\x00\x00\x00d\x01\x00d\x00\x00l\x00\x00}\x01\x00t\x01\x00t\x02\x00\x83\x00\x00\x83\x01\x00|\x00\x00_\x03\x00d\x00\x00S(\x02\x00\x00\x00Ni\xff\xff\xff\xff(\x04\x00\x00\x00t\t\x00\x00\x00itertoolst\x04\x00\x00\x00itert\x03\x00\x00\x00sett\x01\x00\x00\x00x(\x02\x00\x00\x00t\x04\x00\x00\x00selfR\x00\x00\x00\x00(\x00\x00\x00\x00(\x00\x00\x00\x00s\x07\x00\x00\x00<stdin>t\x07\x00\x00\x00breakme\x02\x00\x00\x00s\x04\x00\x00\x00\x00\x01\x0c\x01q\x12\x85q\x13Rq\x14c__builtin__\n__main__\nh\x0fNN}q\x15tq\x16Rq\x17U\x07__doc__q\x18Nutq\x19Rq\x1a)\x81q\x1b}q\x1cb.'
>>> u.breakme()
>>> dill.dumps(u)
Traceback (most recent call last):
…(snip)… 
pickle.PicklingError: Can't pickle <type 'setiterator'>: it's not found as __builtin__.setiterator
>>>

If the error message wasn't good, I could use dill.detectto see which what unpicklable objects the top-level object contains.

如果错误消息不好，我可以使用dill.detect查看顶级对象包含哪些不可选择的对象。

>>> dill.detect.badobjects(u, depth=1)
{'__hash__': <method-wrapper '__hash__' of Unpicklable object at 0x10a37b350>, '__setattr__': <method-wrapper '__setattr__' of Unpicklable object at 0x10a37b350>, '__reduce_ex__': <built-in method __reduce_ex__ of Unpicklable object at 0x10a37b350>, '__reduce__': <built-in method __reduce__ of Unpicklable object at 0x10a37b350>, '__str__': <method-wrapper '__str__' of Unpicklable object at 0x10a37b350>, '__format__': <built-in method __format__ of Unpicklable object at 0x10a37b350>, '__getattribute__': <method-wrapper '__getattribute__' of Unpicklable object at 0x10a37b350>, '__delattr__': <method-wrapper '__delattr__' of Unpicklable object at 0x10a37b350>, 'breakme': <bound method Unpicklable.breakme of <__main__.Unpicklable object at 0x10a37b350>>, '__repr__': <method-wrapper '__repr__' of Unpicklable object at 0x10a37b350>, '__dict__': {'x': <setiterator object at 0x10a370820>}, 'x': <setiterator object at 0x10a370820>, '__sizeof__': <built-in method __sizeof__ of Unpicklable object at 0x10a37b350>, '__init__': <method-wrapper '__init__' of Unpicklable object at 0x10a37b350>}
>>> dill.detect.badtypes(u, depth=1)
{'__hash__': <type 'method-wrapper'>, '__setattr__': <type 'method-wrapper'>, '__reduce_ex__': <type 'builtin_function_or_method'>, '__reduce__': <type 'builtin_function_or_method'>, '__str__': <type 'method-wrapper'>, '__format__': <type 'builtin_function_or_method'>, '__getattribute__': <type 'method-wrapper'>, '__delattr__': <type 'method-wrapper'>, 'breakme': <type 'instancemethod'>, '__repr__': <type 'method-wrapper'>, '__dict__': <type 'dict'>, 'x': <type 'setiterator'>, '__sizeof__': <type 'builtin_function_or_method'>, '__init__': <type 'method-wrapper'>}
>>> set(dill.detect.badtypes(u, depth=1).values())
set([<type 'dict'>, <type 'method-wrapper'>, <type 'instancemethod'>, <type 'setiterator'>, <type 'builtin_function_or_method'>])

dilldoesn't rely on the __getstate__method being present, although maybe it should utilize it if it exists. You can also use objgraphto get a picture of all the object dependencies that are used to build the thing that doesn't pickle. It can help you sort out what the root of the problem is, based on the above information.

dill不依赖于__getstate__存在的方法，尽管如果它存在，它可能应该使用它。您还可以使用objgraph获取用于构建不腌制事物的所有对象依赖关系的图片。它可以帮助您根据上述信息理清问题的根源。

See dill.detectuse in tracking down unpicklable items in this issue: https://github.com/uqfoundation/dill/issues/58

请参阅dill.detect本期用于追踪不可选择的项目：https: //github.com/uqfoundation/dill/issues/58