网页抓取失败，因为请求加载应用时超时
我正在尝试抓取这个网页，但页面一直下载的是一个空白页面，显示如下：
连接错误。
很抱歉，加载应用的请求已超时。
请检查你的网络连接并刷新页面。
如果问题仍然存在，请尝试在不同的浏览器中打开，然后联系支持
刷新页面 →
我想知道我还能做些什么来绕过这个问题。也在想这个问题是否确实与超时有关，如信息所示，这也是我好奇在那个阶段应该尝试哪些“技巧”。
我正在使用PhantomJS，并通过R 进行驱动，代码如下：
library(readr)
library(webdriver)
dr <- run_phantomjs()
ses <- Session$new(port = dr$port)
ses$go("https://virtual.oxfordabstracts.com/#/event/3738/people?page=114")
Sys.sleep(2) # just in case
# save
readr::write_lines(ses$getSource(), "page114.html")
# problem: empty page
解决方案
结论先行
"PhantomJS在过去七年未被维护，因此很可能无法在使用更现代功能的网站上工作。" & PhantomJS自 2018年起已被暂停.
-> 取而代之，我建议使用 {chromote} 或类似 {selenider}（封装了chromote或 selenium）或 rvest::read_html_live，其中使用的是chromote。更好的是：直接对它们的API进行POST/GET请求，因为这可以绕过你遇到的所有问题，且通常更快。研究 margusl的回答 将对你有所启发！
关于你的问题
- 你尝试访问第114页，但最后一页是95页：
即使可以工作，也意义不大。 - 你看到的这是回退页面，当真实内容无法被
<script type="module" crossorigin src="/cdn/virtual-conference/assets/index-BqUYKt9W.js"></script>加载时
我建议改用chromote / selenider的无头浏览器，它们工作正常。然后我仍然怀疑这可能是一个xy问题。这也是我在下方提供替代方案的原因。
旧答案
问题在于加载时的一个很小的初始延迟。
要获取页面的HTML，简单地使用 [rvest::read_html_live]，它在底层使用了chromote。
page <- rvest::read_html_live("https://virtual.oxfordabstracts.com/event/75765/people")
Sys.sleep(2)
page |>
rvest::html_elements("html > *") |>
as.character() |>
paste(collapse = "\n") |>
writeLines("page.html")
但我认为这不太可能给你想要的内容，很可能是参与者及其会议元数据。因此我建议这样做
1) 遍历div并创建一个 data.frame
library(rvest)
html_live <- read_html_live("https://virtual.oxfordabstracts.com/event/75765/people")
Sys.sleep(1.5) # wait for initial page load
# extract cards data
get_cards_data <- function(html_live){
# each author's cards
cards <- html_live |> html_elements("div.shadow-md")
lapply(cards, \(card){
# for each card the author / symposium sections below
childs = card |> html_elements("i.material-icons ~ div")
lapply(childs, \(x) data.frame(
type = x |> html_elements(".text-xs") |> html_text(), # holds presenter / author
event_name = x |> html_elements("a") |> html_text(), # event names come from links
event_link = x |> html_elements("a") |> html_attr("href")
)
) |> do.call(rbind, args = _) |>
transform(
card_title_name = card |> html_elements("h2") |> html_text(trim = TRUE),
affiliation = card |> html_element("i.material-icons ~ div") |> html_text(trim = TRUE)
)
}
) |> do.call(rbind, args = _)
}
# list of resulting tables;
# generally we should avoid growing objects,
# but here it's just a relatively small list of data.frame references
tbls <- list()
# look for the next page link, that's the last child of the nav bar that has a valid href
next_page_selector <- "nav[aria-label='Pagination'] a:last-child[href*='page']"
cli::cli_progress_bar("Collecting card data")
repeat{
tbls <- c(tbls, list(get_cards_data(html_live)))
# stop if next page link can't be found
if (length(html_element(html_live, next_page_selector)) == 0) break
# click and wait for data to be loaded
html_live$click(next_page_selector)
html_live$session$Runtime$evaluate(
"new Promise(r => setTimeout(r, 600))", # you might need to set the timeout to something higher like 1000 ms, because this website loads in content via API calls and JS
awaitPromise = TRUE
)
cli::cli_progress_update()
}
cli::cli_progress_update(force = TRUE)
# ⠇ Collecting card data 49 done (0.9/s) | 54.3s
# bind tbls
final <- tbls |> do.call(rbind, args = _)
head(final)
| 类型 | 活动名称 | 活动链接 | 卡片标题名称 | 所属单位 |
|---|---|---|---|---|
| 作者 | What Elites Believe About Opposing Trump | https://virtual.oxfordabstracts.com/event/75765/submission/340 | Laurits Aarslew | Aarhus University, Denmark |
| 作者 | PO06: Advances in the Study of Elites, Citizens, and Democracy, 10:50-12:30 Friday, 19 June, 2026 | https://virtual.oxfordabstracts.com/event/75765/session/218021 | Laurits Aarslew | Aarhus University, Denmark |
| 会议主席/讨论人 | HI09: Governing Minds: Socialisation, Loyalty, and Regime Survival, 13:30-15:10 Thursday, 18 June, 2026 | https://virtual.oxfordabstracts.com/event/75765/session/218042 | Lasse Aaskoven | University of Southern Denmark |
| 演讲者 | Voting as if your rights depended on it: Understanding the motivations of minority voters | https://virtual.oxfordabstracts.com/event/75765/submission/1293 | M. Belén Abdala | University of Salzburg, Austria |
| 作者 | EL17: Mobilising and Facilitating Turnout, 09:00-10:40 Thursday, 18 June, 2026 | https://virtual.oxfordabstracts.com/event/75765/session/217453 | M. Belén Abdala | University of Salzburg, Austria |
| 演讲者 | Assessing Refugee Rights: New Data and Analysis | https://virtual.oxfordabstracts.com/event/75765/submission/1481 | Lamis Abdelaaty | Syracuse University, USA |
2) 使用 GraphQL API
网站在后台对API进行POST请求，在你浏览页面时获取参与者数据。你可以使用 httr2 重现。步骤如下：
- 打开网站
- 按下
F12 Network->Fetch/XHR- 重新加载
F5 - 查找
graphql-请求 - 如果这是你想要的数据，请查看预览
- 复制载荷
- 在这里找到你的身份验证令牌（Authentication-Token）以及
req_headers()的Content-Type + Origin。注：请求头字段在不同情形下可能有所不同，你可以在graphql -> 复制 -> 以fetch方式复制，便于快速复制请求
- 找到请求的URL与方法
现在构建一些辅助工具来
- 发起你的请求
req - 绑定返回的嵌套列表
rbindlist.v2
然后进行查询并最终合并。请注意： 这只是一个关于概念的示例，你的最终方法可能会有所不同。
# httr2 approach
library(httr2)
auth_token <- "Bearer ..." # your Authentication-Token here. These expire!
# helper that queries "oxfordabstracts" graphql
req <- function(query, tok, operationName)
{
request("https://graphql.oxfordabstracts.com/v1/graphql") |>
req_method("POST") |>
req_headers(
"Content-Type" = "application/json",
"Origin" = "https://virtual.oxfordabstracts.com",
"Authorization" = tok) |>
req_body_json(list(
operationName = operationName,
query = query,
variables = setNames(list(), character(0)) # like {}
)) |>
req_perform() |>
resp_body_json()
}
# helper that flattens nested lists https://stackoverflow.com/a/68001784/28479453
library(foreach)
# this is really preference, you can also choose the purrr::map_dfr and pluck your nested list manually / rowbind / rbind-functions of your choice. Below code works very well for this case
rbindlist.v2 <- function(l)
{
l <- l[lapply(l, class) == "list"]
df <- foreach(element = l, .combine = dplyr::bind_rows, .errorhandling = 'remove') %do%
{df = unlist(element); df = as.data.frame(t(df)); rm(element); return(df)}
rm(l)
return(df)
}
# maybe adjust the limit - in this case it's 2264
'query people_page_search {
ordered_people_by_last_name_v2(
offset: 0
limit: 5000
where: {_and: [{_or: [{presenting_at_least_one: {_eq: true}}, {author_ids: {_neq: []}}, {session_chair_ids: {_neq: []}}, {symposium_chair_ids: {_neq: []}}, {symposium_discussant_ids: {_neq: []}}, {poster_author_ids: {_neq: []}}, {speaker_ids: {_neq: []}}]}]}
) {
author_ids
session_chair_ids
symposium_chair_ids
symposium_discussant_ids
poster_author_ids
speaker_ids
role
first_name
last_name
name
presenting_at_least_one
__typename
}
}' |> req(auth_token, "people_page_search") |>
purrr::pluck("data", "ordered_people_by_last_name_v2") |>
rbindlist.v2() -> p_search
author_ids <- p_search$author_ids |> unlist() |> unique() |> na.omit() |> paste0(collapse = ",")
# here I query only for authors from author_ids in p_search
sprintf('query people_data {
authors(
where: {id: {_in: [%s]}}
) {
id
profile_picture
presenting
user(where: {name_badges: {}}) {
id
name_badges {
avatar
__typename
}
__typename
}
submission {
id
serial_number
program_code
event_id
responses(where: {question: {is_title: {_eq: true}}}) {
value
__typename
}
symposium {
program_sessions_symposia {
start_time
end_time
program_session {
name
id
start_time
end_time
event_id
program_date {
program_date
__typename
}
__typename
}
__typename
}
__typename
}
program_sessions_submissions {
program_session {
name
id
start_time
end_time
event_id
program_date {
program_date
__typename
}
__typename
}
end_time
start_time
__typename
}
__typename
}
affiliations: affiliations_without_hidden_responses {
city
country
institution
state
affiliation_order
__typename
}
__typename
}
}', author_ids) |>
req(auth_token, "people_data") |>
purrr::pluck("data", "authors") |>
rbindlist.v2() -> p_data
# Join
final <- dplyr::left_join(p_search, p_data, by = c("author_ids" = "id"))
你最终会得到一个需要选择/清洗/类型转换的大表。
> str(final)
'data.frame': 2264 obs. of 60 variables:
$ author_ids : chr "5583047" NA "5667156" "5668934" ...
$ role : chr "author" "session_chair" "author" "author" ...
$ first_name : chr "Laurits" "Lasse" "M. Belén" "Lamis" ...
$ last_name : chr "Aarslew" "Aaskoven" "Abdala" "Abdelaaty" ...
$ name : chr "Laurits Aarslew" "Lasse Aaskoven" "M. Belén Abdala" "Lamis Abdelaaty" ...
$ presenting_at_least_one : chr "FALSE" "FALSE" "TRUE" "TRUE" ...
$ __typename.x : chr "ordered_people_by_last_name_v2" "ordered_people_by_last_name_v2" "ordered_people_by_last_name_v2" "ordered_people_by_last_name_v2" ...
$ session_chair_ids : chr NA "98982" NA NA ...
$ symposium_chair_ids : chr NA NA NA NA ...
$ role1 : chr NA NA NA NA ...
$ role2 : chr NA NA NA NA ...
$ symposium_discussant_ids : chr NA NA NA NA ...
$ role3 : chr NA NA NA NA ...
$ author_ids1 : chr NA NA NA NA ...
$ author_ids2 : chr NA NA NA NA ...
$ author_ids3 : chr NA NA NA NA ...
$ author_ids4 : chr NA NA NA NA ...
$ author_ids5 : chr NA NA NA NA ...
$ symposium_discussant_ids1 : chr NA NA NA NA ...
$ symposium_discussant_ids2 : chr NA NA NA NA ...
$ symposium_chair_ids1 : chr NA NA NA NA ...
$ symposium_chair_ids2 : chr NA NA NA NA ...
$ symposium_discussant_ids3 : chr NA NA NA NA ...
$ role4 : chr NA NA NA NA ...
$ author_ids6 : chr NA NA NA NA ...
$ author_ids7 : chr NA NA NA NA ...
$ presenting : chr "FALSE" NA "TRUE" "TRUE" ...
$ submission.id : chr "1185834" NA "1201122" "1201557" ...
$ submission.serial_number : chr "340" NA "1293" "1481" ...
$ submission.event_id : chr "75765" NA "75765" "75765" ...
$ submission.responses.value : chr "<p>What Elites Believe About Opposing Trump</p>" NA "<p>Voting as if your rights depended on it: Understanding the motivations of minority voters </p>" "<p>Assessing Refugee Rights: New Data and Analysis </p>" ...
$ submission.responses.__typename : chr "question_responses" NA "question_responses" "question_responses" ...
$ submission.program_sessions_submissions.program_session.name : chr NA NA "EL17: Mobilising and Facilitating Turnout" NA ...
$ submission.program_sessions_submissions.program_session.id : chr NA NA "217453" NA ...
$ submission.program_sessions_submissions.program_session.start_time : chr NA NA "09:00:00" NA ...
$ submission.program_sessions_submissions.program_session.end_time : chr NA NA "10:40:00" NA ...
$ submission.program_sessions_submissions.program_session.event_id : chr NA NA "75765" NA ...
$ submission.program_sessions_submissions.program_session.program_date.program_date : chr NA NA "2026-06-18" NA ...
$ submission.program_sessions_submissions.program_session.program_date.__typename : chr NA NA "program_dates" NA ...
$ submission.program_sessions_submissions.program_session.__typename : chr NA NA "program_sessions" NA ...
$ submission.program_sessions_submissions.__typename : chr NA NA "program_sessions_submissions" NA ...
$ submission.__typename : chr "submissions" NA "submissions" "submissions" ...
$ affiliations.country : chr "Denmark" NA "Austria" "USA" ...
$ affiliations.institution : chr "Aarhus University" NA "University of Salzburg" "Syracuse University" ...
$ affiliations.affiliation_order : chr "0" NA "0" "0" ...
$ affiliations.__typename : chr "affiliations_without_hidden_responses" NA "affiliations_without_hidden_responses" "affiliations_without_hidden_responses" ...
$ __typename.y : chr "authors" NA "authors" "authors" ...
$ submission.symposium.program_sessions_symposia.program_session.name : chr "PO06: Advances in the Study of Elites, Citizens, and Democracy" NA NA "MI02: Protection, Repression, and Discrimination: Government Responses to Internal and International Displacement" ...
$ submission.symposium.program_sessions_symposia.program_session.id : chr "218021" NA NA "217341" ...
$ submission.symposium.program_sessions_symposia.program_session.start_time : chr "10:50:00" NA NA "10:50:00" ...
$ submission.symposium.program_sessions_symposia.program_session.end_time : chr "12:30:00" NA NA "12:30:00" ...
$ submission.symposium.program_sessions_symposia.program_session.event_id : chr "75765" NA NA "75765" ...
$ submission.symposium.program_sessions_symposia.program_session.program_date.program_date: chr "2026-06-19" NA NA "2026-06-20" ...
$ submission.symposium.program_sessions_symposia.program_session.program_date.__typename : chr "program_dates" NA NA "program_dates" ...
$ submission.symposium.program_sessions_symposia.program_session.__typename : chr "program_sessions" NA NA "program_sessions" ...
$ submission.symposium.program_sessions_symposia.__typename : chr "program_sessions_symposia" NA NA "program_sessions_symposia" ...
$ submission.symposium.__typename : chr "symposia" NA NA "symposia" ...
$ user.id : chr NA NA NA NA ...
$ user.name_badges.__typename : chr NA NA NA NA ...
$ user.__typename : chr NA NA NA NA ...
站内所有文章版权归属LeftHeroAI导航站，无授权禁止任何主体转载、抄袭、复制内容，亦不得私自架设镜像站点。一经侵权，本站将通过法律途径追责。
相关文章
汇集全球数千款 AI 工具，每日持续更新，配套 AI 开发学习资源，轻松用 AI 提升工作效率。快捷键 Ctrl+D/⌘+D 一键收藏本站。