我正在使用Nokogiri抓取事件列表的“ The Broadway Bookshop”,但是页面或多或少都是一个内容块。我似乎无法使用h3
来访问特定的class="news"
项目:
def scrape_broadway_books
base_url = "https://broadwaybookshophackney.com"
slug = "/events/?event=archive"
url = base_url + slug
unparsed_page = HTTParty.get(url)
parsed_page = Nokogiri::HTML(unparsed_page)
events_list = parsed_page.at_css("div#content")
# binding.pry
events = Array.new
events_list.each do |item|
puts item.css("h3.news").text
end
end
这给了我错误:
undefined method `css' for ["id", "content"]:Array (NoMethodError)
如果我尝试遍历events_list
,我仍然只会将一个事件写入我的数据文件中:
events_list = parsed_page.css("div#content")
events = Array.new
events_list.each_with_index do |event, index|
event = {
index: index,
title: event.css("h3.news").text
}
events << event
end
File.open("./_data/events.json", "w") do |file|
file.write(JSON.pretty_generate(events))
end
为什么不能遍历内容div?
[at_css
returns a single element。它不会重复;当您尝试对其进行迭代时,您将获得div的属性。如果要获取要迭代的元素列表,请使用parsed_page.css("div#content")
。
或更简洁地说:
events = parsed_page.css("div#content h3.news").map(&:text)
完成了一点重构之后,您可能还考虑稍微重构该方法,以便您的URL提取和文档解析可重复使用,从而将您的scrape_broadway_books
清除为一个职责:
def scrape_broadway_books
doc = doc_from_url("https://broadwaybookshophackney.com",
"/events/?event=archive")
doc.css("div#content h3.news").map(&:text)
end
def doc_from_url(*parts)
uri = URI.join(*parts)
Nokogiri::HTML(HTTParty.get(uri))
end