我正在尝试从Internet档案馆中获取在一定日期范围内每一天的《华盛顿邮报》首页最后快照的时间。问题是,尽管我收集了日期对象列表,但Selenium并不总是选择正确的日期,而且似乎没有错误。例如,硒将从1月31日跳到2月11日,而不是2月1日。
打印输出:
Moving to 31 Date: 2020-01-31 00:00:00 Last snapshot taken at 22:59:03 Moving to 1 Date: 2020-02-11 00:00:00 Last snapshot taken at 23:53:32 Moving to 2 Date: 2020-02-02 00:00:00 Last snapshot taken at 23:59:56
在此输出中,我们可以看到它应该为1,并且日期对象的文本为1,这意味着该日期对象是从页面中正确提取的。硒没有正确地悬停在它上面吗?
完整代码:
urls = ['https://web.archive.org/web/20190901000000*/washingtonpost.com', 'https://web.archive.org/web/*/washingtonpost.com'] browser = webdriver.Firefox(executable_path='/usr/local/bin/geckodriver') # brew install chromedriver then see in terminal where it was installed to and paste this data = {} for j in range(0, len(urls)): browser.get(urls[j]) calendar_grid = WebDriverWait(browser, 20).until(EC.element_to_be_clickable((By.CLASS_NAME, 'calendar-grid'))) if j == 0: # 2019 browser.execute_script("window.scrollTo(0, document.body.scrollHeight);") time.sleep(10) start = len(dates) - 1 end = 0 step = -1 elif j == 1: # 2020 start = 0 end = len(dates) step = 1 dates = calendar_grid.find_elements_by_css_selector('.calendar-day') print('Dates on page: ' + str(len(dates))) for i in range(start, end, step): if j==0 and len(data) == len(desired_ranges[j]): # to end 2019 break # Hover over the date, let popup appear, wait for loader to disappear, select scroll area print('Moving to ' + dates[i]) hov = ActionChains(browser).move_to_element(dates[i]) hov.perform() popup = WebDriverWait(browser, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, '.popup-of-day-content'))) WebDriverWait(browser, 20).until(EC.invisibility_of_element_located((By.TAG_NAME, 'svg'))) scroll_area = WebDriverWait(browser, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, '.popup-of-day-content > ul > div'))) # Get date and check that it is in our range date = popup.find_element_by_class_name('day-tooltip-title') date_formatted = datetime.datetime.strptime(date.text, '%B %d, %Y') print('Date: ' + str(date_formatted)) if date_formatted not in desired_ranges[j]: continue # skip if it is not else: attempts = 0 while attempts < 5: try: browser.execute_script('arguments[0].scrollTop = arguments[0].scrollHeight', scroll_area) snapshots = popup.find_elements_by_tag_name('a') last_snapshot = snapshots[len(snapshots) - 1] print('Last snapshot taken at ' + last_snapshot.text) data[date_formatted] = {'link': last_snapshot.get_attribute('href'), 'time': last_snapshot.text, 'headlines': []} break except StaleElementReferenceException: attempts += 1
用于设置日期范围的附加代码:
start_day = datetime.date(2019, 12, 8)
end_day = datetime.date.today()
days = (end_day - start_day).days
desired_range = pd.date_range(start_day, periods=days).tolist()
print('Range: ' + str(start_day) + ' to ' + str(end_day))
print('Days: ' + str(days))
desired_range
def time_in_range(start, end, x):
'''Return true if x is in the range [start, end]'''
if start <= end:
return start <= x <= end
else:
return start <= x or x <= end
#Get 2019 date range
desired_range_in_2019 = [x for x in desired_range if time_in_range(datetime.date(2019, 1, 1), datetime.date(2019, 12, 31), x)]
desired_range_in_2020 = [x for x in desired_range if time_in_range(datetime.date(2020, 1, 1), datetime.date(2020, 12, 31), x)]
desired_ranges = [desired_range_in_2019, desired_range_in_2020]
print('Dates in 2019: ' + str(len(desired_range_in_2019)))
print('Dates in 2020: ' + str(len(desired_range_in_2020)))
我正在尝试从Internet档案馆中获取在一定日期范围内每一天的《华盛顿邮报》首页最后快照的时间。问题是硒并不总是选择...
[最有可能发生这种情况是因为ActionChains
模拟了用户的鼠标移动,因此,在到达预期元素的途中,光标(在到达适当元素的途中)会偶然发现其他元素。通过网络节流可以可靠地重现该问题。在Developer Tools中将network更改为Regular 3G