如何在Python的 multiprocessing.Pool中使用多个参数？

编程语言 2026-07-09

我写了一个程序，通过网络爬取来整理数据

我通常通过一个参数来调用它，函数会完成剩下的工作

示例

import multiprocessing

Links = [
        "https://www.[example].com",
        "https://www.[example2].com"

]
def extract_and_compile(link:str, compile_amount:int = 5):
    #because its just an example it doesnt reflect an actual program
    for x in range(compile_amount):
        print(f"compiled data {x+1} from {link}")


if __name__ == "__main__":
    with multiprocessing.Pool() as pool:
        pool.map(extract_and_compile, Links)

上面的这个设置起作用了，因为它产生了一个合适的输出

compiled data 1 from https://www.[example].com
...
compiled data 5 from https://www.[example].com
compiled data 1 from https://www.[example2].com
...
compiled data 5 from https://www.[example2].com

但现在我想通过进程池来改变 "compile_amount"；如果把参数放进可迭代对象中，会在真实程序中把函数搞乱

if __name__ == "__main__":
    with multiprocessing.Pool() as pool:
        pool.map(extract_and_compile, (Links,2))

示例函数返回了一个输出，但正如你所猜，这并不是我想要的输出

compiled data 1 from 2
...
compiled data 1 from ['https://www.[example].com', 'https://www.[example2].com']
...

那么我怎样才能让这个函数的第二个属性通过进程池来改变？

我已经在网上到处查找，但没有找到有用的答案

解决方案

如果你想把同一个 2 传给所有函数，那么你可以使用 functools.partial

from functools import partial

func = partial(extract_and_compile, compile_amount=2)
pool.map(func, Links)

你不能用 lambda，因为 multiprocessing 在发送 lambda 时可能有问题，因为它必须把函数 pickle 发送到另一个进程，而 lambda 不能被pickle。

func = lambda url: extract_and_compile(url, 2)
pool.map(func, Links)  # ERROR

你也可以使用普通函数，但它不能定义在其他函数内部，因为 multiprocessing 在 pickle 上会有问题。

def global_func(url):
    extract_and_compile(url, 2)

pool.map(global_func, Links)

如果你需要发送不同的值，那么你需要创建一个包含元组的列表 (url, val)

Links = [
     ("https://www.[example].com", 2),
     ("https://www.[example2].com", 3)
]

并且需要使用 starmap() 而不是 map()

pool.starmap(func, Links)

如果你使用 map，它会把元组作为一个参数发送，随后你需要把它解包成变量。

def extract_and_compile(data)
    link, compile_amount = data

如果你有参数作为两个列表（长度相同），那么你可以使用 zip() 完成这件事

Links = [
     "https://www.[example].com",
     "https://www.[example2].com",
]

amounts = [2, 3]
#amounts = [2] * len(Links)

pool.starmap(func, zip(Links, amounts))

用于测试的完整代码：

from functools import partial
import multiprocessing

# fmt: off
links = [
    "https://www.[example1].com", 
    "https://www.[example2].com"
]
# fmt: on


def extract_and_compile(link: str, compile_amount: int = 5):
    # because its just an example it doesnt reflect an actual program
    print(f"{link=} | {compile_amount=}")
    # for x in range(compile_amount):
    #    print(f"compiled data {x+1} from {link}")


def version_1():
    print("=== version 1 (original with one parameter) ===")

    with multiprocessing.Pool() as pool:
        pool.map(extract_and_compile, links)


def version_2():
    print("=== version 2 (partial) ===")

    func = partial(extract_and_compile, compile_amount=2)

    with multiprocessing.Pool() as pool:
        pool.map(func, links)


def version_2_lambda():
    print("=== version 2 (lambda) ===")

    func = lambda url: extract_and_compile(url, 2)

    with multiprocessing.Pool() as pool:
        pool.map(func, links)


# can't be defined inside `version_2_def()`
def global_func(url): 
    extract_and_compile(url, 2)

def version_2_def():
    print("=== version 2 (def) ===")

    with multiprocessing.Pool() as pool:
        pool.map(global_func, links)


def version_3():
    print("=== version 3 (tuples + starmap)===")

    # fmt: off
    data = [
        ("https://www.[example1].com", 2),
        ("https://www.[example2].com", 3)
    ]
    # fmt: on

    with multiprocessing.Pool() as pool:
        pool.starmap(extract_and_compile, data)


def version_4():
    print("=== version 4 (zip + starmap) ===")

    amounts = [2, 3]
    # data = zip(links, amounts)

    with multiprocessing.Pool() as pool:
        pool.starmap(extract_and_compile, zip(links, amounts))
        # pool.starmap(extract_and_compile, data)


if __name__ == "__main__":
    # version_1()
    # version_2()
    # version_2_lambda()  # ERROR
    # version_2_def()
    # version_3()
    version_4()

Python