并行运行单独的进程 - Python
声明：本页面是StackOverFlow热门问题的中英对照翻译，遵循CC BY-SA 4.0协议，如果您需要使用它，必须同样遵循CC BY-SA许可，注明原文地址和作者信息，同时你必须将它归于原作者（不是我）：StackOverFlow
原文地址: http://stackoverflow.com/questions/19080792/
Warning: these are provided under cc-by-sa 4.0 license. You are free to use/share it, But you must attribute it to the original authors (not me):
StackOverFlow
Run separate processes in parallel - Python
提问by Bade
I use python 'multiprocessing' module to run single process on multiple cores but I want to run a couple of independent process in parallel. For ex - Process one parses large file, Process two find pattern in different file and process three does some calculation; can all three different processed that have different set of arguments be run in parallel?
我使用 python 'multiprocessing' 模块在多个内核上运行单个进程，但我想并行运行几个独立进程。例如 - 进程一解析大文件，进程二在不同的文件中查找模式，进程三做一些计算；可以并行运行具有不同参数集的所有三个不同处理吗？
def Process1(largefile):
Parse large file
runtime 2hrs
return parsed_file
def Process2(bigfile)
Find pattern in big file
runtime 2.5 hrs
return pattern
def Process3(integer)
Do astronomical calculation
Run time 2.25 hrs
return calculation_results
def FinalProcess(parsed,pattern,calc_results):
Do analysis
Runtime 10 min
return final_results
def main():
parsed = Process1(largefile)
pattern = Process2(bigfile)
calc_res = Process3(integer)
Final = FinalProcess(parsed,pattern,calc_res)
if __name__ == __main__:
main()
sys.exit()
In above pseudo code Process1, Process2 and Process3 are single core process i.e they can't be run on multiple processors. These processes are run sequentially and take 2+2.5+2.25hrs = 6.75 hrs. Is it possible to run these three process in parallel? So that they run at same time on different processors/cores and when most time taking (Process2)finishes than we move to Final Process.
在上面的伪代码中，Process1、Process2 和 Process3 是单核进程，即它们不能在多个处理器上运行。这些进程按顺序运行，需要 2+2.5+2.25hrs = 6.75 小时。是否可以并行运行这三个进程？这样它们就可以在不同的处理器/内核上同时运行，并且当大部分时间 (Process2) 完成时，我们会转到最终进程。
I would really appreciate your help.
我将衷心感谢您的帮助。
AK
AK
采纳答案by leon
From 16.6.1.5. Using a pool of workers:
from multiprocessing import Pool
def f(x):
return x*x
if __name__ == '__main__':
pool = Pool(processes=4) # start 4 worker processes
result = pool.apply_async(f, [10]) # evaluate "f(10)" asynchronously
print result.get(timeout=1) # prints "100" unless your computer is *very* slow
print pool.map(f, range(10)) # prints "[0, 1, 4,..., 81]"
You can, therefore, apply_async against a pool and get your results after everything is ready.
因此，您可以对池 apply_async 并在一切准备就绪后获得结果。
from multiprocessing import Pool
# all your methods declarations above go here
# (...)
def main():
pool = Pool(processes=3)
parsed = pool.apply_async(Process1, [largefile])
pattern = pool.apply_async(Process2, [bigfile])
calc_res = pool.apply_async(Process3, [integer])
pool.close()
pool.join()
final = FinalProcess(parsed.get(), pattern.get(), calc_res.get())
# your __main__ handler goes here
# (...)
