我有一个网络刮板,我很满意,除了有时它错过迭代,因为它没有完全加载网页(这是我正在抓取的网站的性质。在这些情况下,我希望我的代码尝试再次迭代。目前,我的代码框架看起来像这样:
data = []
for i in range(len(links)):
try:
driver.get(link[i])
a = driver.find_elements_by_xpath(#data in here)[0].text
data.append(a)
#this is then written to a csv
except:
print(i)
所以目前,我的代码运行,然后只为我列出哪些数字实例失败。然后我回去手动输入数据。如果不是这样做,我的程序会再次尝试失败的实例,这样我就不会错过数据了。我能用这种方式做到吗?谢谢
如果你想多次重试相同的link[i]
,你可能需要一个额外的循环。究竟什么样的循环取决于一些细节。如果你想继续尝试直到你成功(假设你可以肯定最终会发生),那么while True
循环将是最有意义的。另一方面,如果要限制尝试次数,则范围上的for
循环会更好。
这是一个最多尝试三次的实现草图:
max_tries = 3
data = []
for i, link in enumerate(links): # this is a slightly nicer way to do your main loop
for t in range(max_tries):
try:
driver.get(link)
a = driver.find_elements_by_xpath("#data in here")[0].text
data.append(a)
break # break out of the inner loop if we succeeded
except:
print("failed to load link", i, "retrying..." if t < max_tries-1 else "giving up.")
您可以实现一个迭代计数器,并在第一次尝试后找出两个列表之间的差异,为您的心情:)
data = []
intData = []
counter = 0
maxIterations = 2
def Diff(li1, li2):
return (list(set(li1) - set(li2)))
while counter < maxIterations:
for i in range(len(links)):
try:
if counter < 1:
driver.get(link[i])
a = driver.find_elements_by_xpath(#xpathstring)[0].text
data.append(a)
else:
driver.get(link[i])
a = driver.find_elements_by_xpath(#xpathstring)[0].text
intData.append(a)
counter += 1
except:
print(i)
counter += 1
# Find differences between first iterations and all consecutive ones
print(Diff(intData, data))