Python 多处理：使用 tqdm 显示进度条
声明：本页面是StackOverFlow热门问题的中英对照翻译，遵循CC BY-SA 4.0协议，如果您需要使用它，必须同样遵循CC BY-SA许可，注明原文地址和作者信息，同时你必须将它归于原作者（不是我）：StackOverFlow
原文地址: http://stackoverflow.com/questions/41920124/
Warning: these are provided under cc-by-sa 4.0 license. You are free to use/share it, But you must attribute it to the original authors (not me):
StackOverFlow
Multiprocessing : use tqdm to display a progress bar
提问by SciPy
To make my code more "pythonic" and faster, I use "multiprocessing" and a map function to send it a) the function and b) the range of iterations.
为了使我的代码更加“pythonic”和更快，我使用“multiprocessing”和一个 map 函数来发送它 a) 函数和 b) 迭代范围。
The implanted solution (i.e., call tqdm directly on the range tqdm.tqdm(range(0, 30)) does not work with multiprocessing (as formulated in the code below).
植入的解决方案（即，直接在范围 tqdm.tqdm(range(0, 30)）上调用 tqdm 不适用于多处理（如下面的代码所示）。
The progress bar is displayed from 0 to 100% (when python reads the code?) but it does not indicate the actual progress of the map function.
进度条显示从0到100%（python读取代码的时候？）但并不表示map函数的实际进度。
How to display a progress bar that indicates at which step the 'map' function is ?
如何显示指示“地图”功能在哪一步的进度条？
from multiprocessing import Pool
import tqdm
import time
def _foo(my_number):
square = my_number * my_number
time.sleep(1)
return square
if __name__ == '__main__':
p = Pool(2)
r = p.map(_foo, tqdm.tqdm(range(0, 30)))
p.close()
p.join()
Any help or suggestions are welcome...
欢迎任何帮助或建议...
采纳答案by SciPy
Solution Found : Be careful! Due to multiprocessing, estimation time (iteration per loop, total time, etc.) could be unstable, but the progress bar works perfectly.
找到的解决方案：小心！由于多处理，估计时间（每个循环的迭代次数、总时间等）可能不稳定，但进度条工作正常。
Note: Context manager for Pool is only available from Python version 3.3
注意：池的上下文管理器仅适用于 Python 3.3 版
from multiprocessing import Pool
import time
from tqdm import *
def _foo(my_number):
square = my_number * my_number
time.sleep(1)
return square
if __name__ == '__main__':
with Pool(processes=2) as p:
max_ = 30
with tqdm(total=max_) as pbar:
for i, _ in enumerate(p.imap_unordered(_foo, range(0, max_))):
pbar.update()
回答by hkyi
Use imap instead of map, which returns an iterator of processed values.
使用 imap 而不是 map，它返回一个处理值的迭代器。
from multiprocessing import Pool
import tqdm
import time
def _foo(my_number):
square = my_number * my_number
time.sleep(1)
return square
if __name__ == '__main__':
with Pool(2) as p:
r = list(tqdm.tqdm(p.imap(_foo, range(30)), total=30))
回答by Victor Quach
You can use p_tqdminstead.
你可以p_tqdm改用。
https://github.com/swansonk14/p_tqdm
https://github.com/swansonk14/p_tqdm
from p_tqdm import p_map
import time
def _foo(my_number):
square = my_number * my_number
time.sleep(1)
return square
if __name__ == '__main__':
r = p_map(_foo, list(range(0, 30)))
回答by casper.dcl
Sorry for being late but if all you need is a concurrent map, the latest version (tqdm>=4.42.0) now has this built-in:
抱歉迟到了，但如果您只需要一个并发映射，那么最新版本 ( tqdm>=4.42.0) 现在具有以下内置功能：
from tqdm.contrib.concurrent import process_map # or thread_map
import time
def _foo(my_number):
square = my_number * my_number
time.sleep(1)
return square
if __name__ == '__main__':
r = process_map(_foo, range(0, 30), max_workers=2)
References: https://tqdm.github.io/docs/contrib.concurrent/and https://github.com/tqdm/tqdm/blob/master/examples/parallel_bars.py
参考资料：https: //tqdm.github.io/docs/contrib.concurrent/和https://github.com/tqdm/tqdm/blob/master/examples/parallel_bars.py
回答by Oliver Wilken
based on the answer of Xavi Martínez I wrote the function imap_unordered_bar. It can be used in the same way as imap_unorderedwith the only difference that a processing bar is shown.
根据 Xavi Martínez 的回答，我编写了函数imap_unordered_bar。它的使用方式imap_unordered与显示处理条的唯一区别相同。
from multiprocessing import Pool
import time
from tqdm import *
def imap_unordered_bar(func, args, n_processes = 2):
p = Pool(n_processes)
res_list = []
with tqdm(total = len(args)) as pbar:
for i, res in tqdm(enumerate(p.imap_unordered(func, args))):
pbar.update()
res_list.append(res)
pbar.close()
p.close()
p.join()
return res_list
def _foo(my_number):
square = my_number * my_number
time.sleep(1)
return square
if __name__ == '__main__':
result = imap_unordered_bar(_foo, range(5))
回答by dkrynicki
import multiprocessing as mp
import tqdm
some_iterable = ...
def some_func():
# your logic
...
if __name__ == '__main__':
with mp.Pool(mp.cpu_count()-2) as p:
list(tqdm.tqdm(p.imap(some_func, iterable), total=len(iterable)))
回答by Vijayabhaskar J
This approach simple and it works.
这种方法简单而且有效。
from multiprocessing.pool import ThreadPool
import time
from tqdm import tqdm
def job():
time.sleep(1)
pbar.update()
pool = ThreadPool(5)
with tqdm(total=100) as pbar:
for i in range(100):
pool.apply_async(job)
pool.close()
pool.join()
