Selenium与 BeautifulSoup 4的组合导致.find_all无法正常工作，从而导致变量未定义

编程语言 2026-07-08

我正在把Selenium与我用BeautifulSoup4编写的代码结合起来，用来抓取一个页面上的价格，尽管仅使用BeautifulSoup4时代码运行正常，但在实现Selenium后，.find_all停止工作并让其中一个变量未定义，尽管有要追加的值。我的目标是让所有价格都打印出来并加入到列表中，随后将打印所有价格的合并结果。

这是我仅使用BeautifulSoup时的代码：

from bs4 import BeautifulSoup
import requests
import sched, time



pets = [] 

petsurl = "https://starpets.gg/adopt-me/shop/pet/hot_doggo/17996"
petsresult = requests.get(petsurl)
petdoc = BeautifulSoup(petsresult.text, "html.parser")


pettag1 = petdoc.select("p._text_j98bt_1._text__size_2xl_j98bt_55._text__nowrap_j98bt_7._text__weight_bold_j98bt_83._text__style_normal_j98bt_95._text__decoration_normal_j98bt_104._info__price__count_ilro6_87, span.itemprop.price") 
for petbestPrice in pettag1:
     petprice1 = float(petbestPrice.text.replace("$", ""))
     pets.append(petprice1) 
     print("The best price is:", petprice1)

pettag2 = petdoc.find_all("div", class_ = "_text_j98bt_1 _text__size_m_j98bt_40 _text__weight_bold_j98bt_83 _text__style_normal_j98bt_95 _text__decoration_normal_j98bt_104 _content-price_1xjd5_90")
for petotherPrice in pettag2:
    petprice2 = float(petotherPrice.text.split(' ')[0])
    pets.append(petprice2)
    print(petprice2)

if len(pets) == 0:
     pets.append(0.00)
     print("An error has occurred loading")
else:
     print(pets)

这段代码按预期工作，因为它打印了第一个变量中的值，而第二个变量里打印了所有值，最后整个列表也被打印出来。然而，在加入Selenium之后，只有第一个变量能够正确打印：

from bs4 import BeautifulSoup
import sched, time
from selenium import webdriver
from selenium.webdriver import Chrome
from selenium.webdriver.common.by import By

pets = [] 

driver= Chrome()

driver.get("https://starpets.gg/adopt-me/shop/pet/hot_doggo/17996")
driver.implicitly_wait(5)
cookiedecl = driver.find_element(By.CLASS_NAME, "cookie__button._button_1fum3_1._button__secondary_1fum3_413._button_mm_1fum3_134._button_fullWidth_1fum3_268._button__text_upper_1fum3_73._button__text_ellipsis_1fum3_76.cookie__button").click()
time.sleep(5)
PetAges = driver.find_element(By.CLASS_NAME, "_container_1dlh7_1._size__xl_1dlh7_72")
PetMenu = PetAges.find_element(By.XPATH, '//*[@title="Select age"]').click()
time.sleep(5)
NewbFind = driver.find_element(By.CLASS_NAME, "_container_wdj39_1._size__s_wdj39_87")
NewbClick = NewbFind.find_element(By.XPATH, '//*[text()="Newborn"]').click()
Petdoc = BeautifulSoup(driver.page_source, "html.parser")
time.sleep(10)
Pettag1 = Petdoc.select("p._text_j98bt_1._text__size_2xl_j98bt_55._text__nowrap_j98bt_7._text__weight_bold_j98bt_83._text__style_normal_j98bt_95._text__decoration_normal_j98bt_104._info__price__count_ilro6_87, span.itemprop.price") 
for PetbestPrice in Pettag1: 
 Petprice1 = float(PetbestPrice.text.replace("$", "")) 
 pets.append(Petprice1) 
print(Petprice1)

Pettag2 = Petdoc.find_all("div", class_ = "_text_j98bt_1 _text__size_m_j98bt_40 _text__weight_bold_j98bt_83 _text__style_normal_j98bt_95 _text__decoration_normal_j98bt_104 _content-price_1xjd5_90")
for PetotherPrice in Pettag2:
 Petprice2 = float(PetotherPrice.text.split(' ')[0])
 pets.append(Petprice2)
print(Petprice2)
print(pets)
driver.close()

执行这段代码时，在尝试执行 print(Petprice2) 时会出现错误 NameError: name 'Petprice2' is not defined. Did you mean: 'Petprice1'?。

我尝试在把页面源代码交给BeautifulSoup以防加载问题时使用 time.sleep(10)，但它仍然只打印出第一个价格。

我也尝试把 Petprice2 = float(PetotherPrice.text.split(' ')[0]) 替换为 Petprice2 = float(PetotherPrice.text.replace("$", ""))，但结果一样。

为什么在引入Selenium之后会出现这个问题？有没有办法修复它（并在这个过程中让我的代码变得更好？）

解决方案

当 Pettag2 为空时，它不会运行循环 for PetotherPrice in Pettag2:，也不会运行第 Petprice2 = ... 行，因此不会创建这个变量——这在你尝试在循环之后显示 Petprice2 时会带来问题。

同样的问题也可能出现在 Petprice1，当 Pettag1 为空时。

但把它放在循环内部显示更有意义——因此需要缩进。
并且这样问题就解决了。

如果使用4 个空格进行缩进，效果会更明显。

# - before first loop -

Pettag1 = Petdoc.select("p._text_j98bt_1._text__size_2xl_j98bt_55._text__nowrap_j98bt_7._text__weight_bold_j98bt_83._text__style_normal_j98bt_95._text__decoration_normal_j98bt_104._info__price__count_ilro6_87, span.itemprop.price") 

# - first loop -

for PetbestPrice in Pettag1: 
    Petprice1 = float(PetbestPrice.text.replace("$", "")) 
    pets.append(Petprice1) 
    print("Petprice1:", Petprice1)  # <-- indentation

# - after first loop / before second loop ---

Pettag2 = Petdoc.find_all("div", class_ = "_text_j98bt_1 _text__size_m_j98bt_40 _text__weight_bold_j98bt_83 _text__style_normal_j98bt_95 _text__decoration_normal_j98bt_104 _content-price_1xjd5_90")

# - second loop - 

for PetotherPrice in Pettag2:
    Petprice2 = float(PetotherPrice.text.split(' ')[0])
    pets.append(Petprice2)
    print("Petprice2:", Petprice2)  # <-- indentation

# - after second loop -

print(pets)

我的完整可运行代码，选择器被简化且未使用 BeautifulSoup

# from bs4 import BeautifulSoup
import time
from selenium import webdriver
from selenium.webdriver import Chrome
from selenium.webdriver.common.by import By

pets = []

try:

    driver = Chrome()
    # driver.maximize_window()

    driver.get("https://starpets.gg/adopt-me/shop/pet/hot_doggo/17996")
    driver.implicitly_wait(15)  # and code may no need `time.sleep()`

    # --- cookies ---

    print("cookies")

    # cookiedecl = driver.find_element(
    #     By.CLASS_NAME,
    #     "cookie__button._button_1fum3_1._button__secondary_1fum3_413._button_mm_1fum3_134._button_fullWidth_1fum3_268._button__text_upper_1fum3_73._button__text_ellipsis_1fum3_76.cookie__button",
    # ).click()

    driver.find_element(
        By.XPATH, '//button[contains(@class, "cookie__button")][2]'
    ).click()

    # time.sleep(5)

    # --- search ---

    print("search age")

    # pet_ages = driver.find_element(
    #    By.CLASS_NAME, "_container_1dlh7_1._size__xl_1dlh7_72"
    # )
    # pet_ages.find_element(By.XPATH, '//*[@title="Select age"]').click()

    driver.find_element(By.XPATH, '//button[@title="Select age"]').click()

    # time.sleep(2)

    print("newborn")

    # new_find = driver.find_element(
    #     By.CLASS_NAME, "_container_wdj39_1._size__s_wdj39_87"
    # )
    # new_find.find_element(By.XPATH, '//*[text()="Newborn"]').click()

    driver.find_element(
        By.XPATH, '//div[@role="listitem" and div[div[div[text()="Newborn"]]] ]'
    ).click()

    # time.sleep(5)

    # --- pets ---

    # soup = BeautifulSoup(driver.page_source, "html.parser")

    print("--- loop 1 ---")

    # pet_tags = soup.select(
    #     "p._text_j98bt_1._text__size_2xl_j98bt_55._text__nowrap_j98bt_7._text__weight_bold_j98bt_83._text__style_normal_j98bt_95._text__decoration_normal_j98bt_104._info__price__count_ilro6_87, span.itemprop.price"
    # )

    pet_tags = driver.find_elements(
        By.XPATH, "//div[contains(@class, '_top')]//*[@itemprop='price']"
    )

    print("len:", len(pet_tags))

    for item in pet_tags:
        pet_price = float(item.text.replace("$", ""))
        pets.append(pet_price)
        print("[loop 1] pet_price:", pet_price)

    print("--- loop 2 ---")

    # pet_tags = soup.find_all(
    #     "div",
    #     class_="_text_j98bt_1 _text__size_m_j98bt_40 _text__weight_bold_j98bt_83 _text__style_normal_j98bt_95 _text__decoration_normal_j98bt_104 _content-price_1xjd5_90",
    # )

    pet_tags = driver.find_elements(
        By.XPATH, '//div[contains(@class, "_content-price")]'
    )

    print("len:", len(pet_tags))

    for item in pet_tags:
        pet_price = float(item.text.split(" ")[0])
        pets.append(pet_price)
        print("[loop 2] pet_price:", pet_price)

    print(pets)

except Exception as ex:
    print("Ex:", ex)

input("\nPress ENTER to close")
driver.close()

结果（价格以欧元计）

cookies
search age
newborn
--- loop 1 ---
len: 1
[loop 1] pet_price: 14.43
--- loop 2 ---
len: 8
[loop 2] pet_price: 14.43
[loop 2] pet_price: 14.43
[loop 2] pet_price: 14.43
[loop 2] pet_price: 14.43
[loop 2] pet_price: 14.53
[loop 2] pet_price: 14.67
[loop 2] pet_price: 15.61
[loop 2] pet_price: 16.27
[14.43, 14.43, 14.43, 14.43, 14.43, 14.53, 14.67, 15.61, 16.27]

Press ENTER to close

Python