我在学习RAG（检索增强生成）以及书中提到的下面这段代码。出现错误，所以我添加了agent相关的设置，但仍然没有成功。

from langchain_community.document_loaders import AsyncHtmlLoader

#This is the URL of the Wikipedia page on the 2023 Cricket World Cup


url="https://en.wikipedia.org/wiki/2023_Cricket_World_Cup"

loader = AsyncHtmlLoader(url)

html_data = loader.load()

print(html_data)

输出：

Fetching pages: 100%|####################################################################| 1/1 [00:00<00:00,  3.47it/s]

[Document(metadata={'source': 'https://en.wikipedia.org/wiki/2023_Cricket_World_Cup'}, page_content='Please respect our robot policy https://w.wiki/4wJS when crawling us. Contact [email protected] if you need higher volumes. (b74d0e2) (30224bb)')]

解决方案

当我运行你的脚本时，在 Fetching pages 之前我也看到

USER_AGENT environment variable not set, consider setting it to identify your requests.

错误显示链接 https://w.wiki/4wJS，它描述了如何遵守维基百科的机器人策略。

第2点与头部 USER-AGENT 有关，并且有一个指向 另一个页面 的链接，该页面用Python给出了带有 requests 的示例

import requests

url = 'https://example/...'
headers = {'User-Agent': 'CoolBot/0.0 (https://example.org/coolbot/; [email protected])'}

response = requests.get(url, headers=headers)

当我把它和你的URL一起使用时，我就能得到预期的数据。

而且 AsyncHtmlLoader 也在我把同样的 User-Agent 设置为环境变量，或者直接在Linux上运行时对我起作用，如下所示

USER_AGENT='CoolBot/0.0 (https://example.org/coolbot/; [email protected])' ./main.py

看起来你也可以在代码中使用 header_template= 来设置它，但这可能需要获取 default_header_template 才能实现

from langchain_community.document_loaders import AsyncHtmlLoader
from langchain_community.document_loaders.async_html import default_header_template

headers = dict(default_header_template)
headers["User-Agent"] = "CoolBot/0.0 (https://example.org/coolbot/; [email protected])"

url = "https://en.wikipedia.org/wiki/2023_Cricket_World_Cup"
loader = AsyncHtmlLoader(url, header_template=headers)

all_results = loader.load()

for one_result in all_results:
    print(one_result.page_content)

用于测试的完整代码：

#!/usr/bin/env python3

def test_langchain(url):
    # import os
    # print("[DEBUG] USER_AGENT:", os.getenv("USER_AGENT"))

    from langchain_community.document_loaders import AsyncHtmlLoader

    # This is the URL of the Wikipedia page on the 2023 Cricket World Cup
    from langchain_community.document_loaders.async_html import default_header_template

    #headers = dict(default_header_template)
    headers = default_header_template.copy()

    #print('--- BEFORE ---')
    #for key, value in headers.items():
    #    print(f"{key}: {value}")

    headers["User-Agent"] = "CoolBot/0.0 (https://example.org/coolbot/; [email protected])"

    #print('--- AFTER ---')
    #for key, value in headers.items():
    #    print(f"{key}: {value}")

    loader = AsyncHtmlLoader(url, header_template=headers)
    all_results = loader.load()

    for one_result in all_results:
        print(one_result.page_content)


def test_requests(url):
    import requests

    headers = {"User-Agent": "CoolBot/0.0 (https://example.org/coolbot/; [email protected])"}

    response = requests.get(url, headers=headers)
    print(response.text)


# --- main ---

url = "https://en.wikipedia.org/wiki/2023_Cricket_World_Cup"
# url = "https://httpbin.org/get"  # useful for checking all information in requests (headers, cookies, etc.)

# test_requests(url)
test_langchain(url)