Python 如何让 Pool.map 采用 lambda 函数

声明:本页面是StackOverFlow热门问题的中英对照翻译,遵循CC BY-SA 4.0协议,如果您需要使用它,必须同样遵循CC BY-SA许可,注明原文地址和作者信息,同时你必须将它归于原作者(不是我):StackOverFlow 原文地址: http://stackoverflow.com/questions/4827432/
Warning: these are provided under cc-by-sa 4.0 license. You are free to use/share it, But you must attribute it to the original authors (not me): StackOverFlow

提示:将鼠标放在中文语句上可以显示对应的英文。显示中英文
时间:2020-08-18 17:41:38  来源:igfitidea点击:

How to let Pool.map take a lambda function

pythonmultiprocessingpool

提问by Peter Smit

I have the following function:

我有以下功能:

def copy_file(source_file, target_dir):
    pass

Now I would like to use multiprocessingto execute this function at once:

现在我想用multiprocessing一次执行这个函数:

p = Pool(12)
p.map(lambda x: copy_file(x,target_dir), file_list)

The problem is, lambda's can't be pickled, so this fails. What is the most neat (pythonic) way to fix this?

问题是,lambda 不能被腌制,所以这失败了。解决此问题的最简洁(pythonic)方法是什么?

采纳答案by Fred Foo

Use a function object:

使用函数对象:

class Copier(object):
    def __init__(self, tgtdir):
        self.target_dir = tgtdir
    def __call__(self, src):
        copy_file(src, self.target_dir)

To run your Pool.map:

运行你的Pool.map

p.map(Copier(target_dir), file_list)

回答by unutbu

For Python2.7+or Python3, you could use functools.partial:

对于 Python2.7+或 Python3,您可以使用functools.partial

import functools
copier = functools.partial(copy_file, target_dir=target_dir)
p.map(copier, file_list)

回答by petRUShka

Question is a bit old but if you are still use Python 2 my answer can be useful.

问题有点老了,但如果您仍在使用 Python 2,我的回答可能会很有用。

Trick is to use part of pathosproject: multiprocessfork of multiprocessing. It get rid of annoying limitation of original multiprocess.

技巧是使用pathos项目的一部分:进程的多进程分支。它摆脱了原始多进程的恼人限制。

Installation: pip install multiprocess

安装: pip install multiprocess

Usage:

用法:

>>> from multiprocess import Pool
>>> p = Pool(4)
>>> print p.map(lambda x: (lambda y:y**2)(x) + x, xrange(10))
[0, 2, 6, 12, 20, 30, 42, 56, 72, 90]

回答by Rufus

From thisanswer, pathos let's you run your lambda p.map(lambda x: copy_file(x,target_dir), file_list)directly, saving all the workarounds / hacks

这个答案,pathos 让你p.map(lambda x: copy_file(x,target_dir), file_list)直接运行你的 lambda ,节省所有的解决方法/黑客