带有悲情的 Python 多处理
声明：本页面是StackOverFlow热门问题的中英对照翻译，遵循CC BY-SA 4.0协议，如果您需要使用它，必须同样遵循CC BY-SA许可，注明原文地址和作者信息，同时你必须将它归于原作者（不是我）：StackOverFlow
原文地址: http://stackoverflow.com/questions/26059764/
Warning: these are provided under cc-by-sa 4.0 license. You are free to use/share it, But you must attribute it to the original authors (not me):
StackOverFlow
Python multiprocessing with pathos
提问by user3708829
I am trying to use Python's pathos to designate computations into separate processes in order to accelerate it with multicore processor. My code is organized like:
我正在尝试使用 Python 的 pathos 将计算指定为单独的进程，以便使用多核处理器对其进行加速。我的代码组织如下：
class:
def foo(self,name):
...
setattr(self,name,something)
...
def boo(self):
for name in list:
self.foo(name)
As I had pickling problems with multiprocessing.Pool, I decided to try pathos. I tried, as suggested in previous topics:
由于我在 multiprocessing.Pool 中遇到了酸洗问题，我决定尝试一下 pathos。我尝试过，正如之前主题中所建议的：
import pathos.multiprocessing
but it resulted in error: No module multiprocessing - which I can't find in latest pathos version.
但它导致错误：没有模块多处理 - 我在最新的 pathos 版本中找不到。
Then I tried modify boo method:
然后我尝试修改 boo 方法：
def boo(self):
import pathos
pathos.pp_map.pp_map(self.foo,list)
Now there is no error thrown, but foo does not work - instance of my class has no new attributes. Please help me, because I have no idea where to move next, after a day spent on that.
现在没有抛出错误，但 foo 不起作用 - 我的类的实例没有新属性。请帮助我，因为在花了一天的时间之后，我不知道下一步该往哪里移动。
回答by Mike McKerns
I'm the pathosauthor. I'm not sure what you want to do from your code above.
However, I can maybe shed some light. Here's some similar code:
我是pathos作者。我不确定你想从上面的代码中做什么。但是，我也许可以阐明一些观点。下面是一些类似的代码：
>>> from pathos.multiprocessing import ProcessingPool
>>> class Bar:
... def foo(self, name):
... return len(str(name))
... def boo(self, things):
... for thing in things:
... self.sum += self.foo(thing)
... return self.sum
... sum = 0
...
>>> b = Bar()
>>> results = ProcessingPool().map(b.boo, [[12,3,456],[8,9,10],['a','b','cde']])
>>> results
[6, 4, 5]
>>> b.sum
0
So what happens above, is that the boomethod of the Barinstance bis called where b.boois passed to a new python process, and then evaluated for each of the nested lists. You can see that the results are correct… len("12")+len("3")+len("456") is 6, and so on.
所以上面发生的是，实例的boo方法被调用， where被传递给一个新的 python 进程，然后对每个嵌套列表进行评估。可以看到结果是正确的……len("12")+len("3")+len("456")为6，以此类推。Barbb.boo
However, you can also see that when you look at b.sum, it's mysteriously still 0. Why is b.sumstill zero? Well, what multiprocessing(and thus also pathos.multiprocessing) does, is make a COPYof whatever you pass through the map to the other python process… and then the copied instance is then called (in parallel) and return whatever results are called by the method invoked. Note you have to RETURNresults, or print them, or log them, or send them to a file, or otherwise. They can't go back to the original instance as you might expect, because it's not the original instance that's sent over to the other processors. The copies of the instance are created, then disposed of -- each of them had their sumattribute increased, but the original `b.sum' is untouched.
然而，你也可以看到，当你看的时候b.sum，它神秘地静止了0。为什么b.sum还是零？好吧，multiprocessing（因此也是pathos.multiprocessing）所做的，是将您通过映射传递到另一个 python 进程的任何内容进行COPY……然后（并行）调用复制的实例并返回调用的方法调用的任何结果。请注意，您必须返回结果，或打印它们，或记录它们，或将它们发送到文件，或以其他方式。它们不能像您期望的那样返回到原始实例，因为它不是发送到其他处理器的原始实例。实例的副本被创建，然后被处理——他们每个人都有自己的sum属性增加，但原始的“b.sum”未受影响。
There is however, plans within pathosto make something like the above work as you might expect -- where the original object ISupdated, but it doesn't work like that yet.
然而有，内计划pathos使像上述工作正如您所料-在原来的对象IS更新，但它不工作，这样的呢。
EDIT:If you are installing with pip, note that the latest released version of pathosis several years old, and may not install correctly, or may not install all of the submodules. A new pathosrelease is pending, but until then, it's better to get the latest version of the code from github, and install from there. The trunk is for the most part stable under development. I think your issue may have been that not all packages were installed, due to a "new" pip-- "old" pathosincompatibility in the install. If pathos.multiprocessingis missing, this is the most likely culprit.
编辑：如果您使用 安装pip，请注意最新发布的版本已经pathos有好几年了，可能无法正确安装，或者可能无法安装所有子模块。新pathos版本正在等待中，但在那之前，最好从 github 获取最新版本的代码，然后从那里安装。主干大部分是稳定的，正在开发中。我认为您的问题可能是由于安装中的“新” pip-“旧”pathos不兼容，并非所有软件包都已安装。如果pathos.multiprocessing缺少，这是最有可能的罪魁祸首。
Get pathosfrom github here: https://github.com/uqfoundation/pathos
pathos从 github获取：https: //github.com/uqfoundation/pathos
