ProcessPoolExecutor from concurrent.futures比multiprocessing.Pool慢得多。

7 浏览
0 Comments

ProcessPoolExecutor from concurrent.futures比multiprocessing.Pool慢得多。

我正在尝试使用Python 3.2引入的新的闪亮的concurrent.futures模块进行实验，并且我注意到，几乎相同的代码下，使用concurrent.futures中的Pool比使用multiprocessing.Pool要慢得多。

这是使用multiprocessing的版本：

def hard_work(n):
    # 真正的辛苦工作在这里
    pass
if __name__ == '__main__':
    from multiprocessing import Pool, cpu_count
    try:
        workers = cpu_count()
    except NotImplementedError:
        workers = 1
    pool = Pool(processes=workers)
    result = pool.map(hard_work, range(100, 1000000))

而这是使用concurrent.futures的版本：

def hard_work(n):
    # 真正的辛苦工作在这里
    pass
if __name__ == '__main__':
    from concurrent.futures import ProcessPoolExecutor, wait
    from multiprocessing import cpu_count
    try:
        workers = cpu_count()
    except NotImplementedError:
        workers = 1
    pool = ProcessPoolExecutor(max_workers=workers)
    result = pool.map(hard_work, range(100, 1000000))

使用从这篇Eli Bendersky文章中获取的一个简单的因式分解函数，这是在我的电脑上（i7，64位，Arch Linux）的结果：

[juanlu@nebulae]─[~/Development/Python/test]
└[10:31:10] $ time python pool_multiprocessing.py 
real    0m10.330s
user    1m13.430s
sys 0m0.260s
[juanlu@nebulae]─[~/Development/Python/test]
└[10:31:29] $ time python pool_futures.py 
real    4m3.939s
user    6m33.297s
sys 0m54.853s

由于我遇到了pickle错误，无法使用Python分析器对其进行分析。有什么建议吗？

0