我经常需要在供应商的安全公告页面上列出CVE的列表。有时,复制起来很简单,但通常将它们与一堆文本混在一起。
[我已经有一段时间没有接触过Python了,所以我认为这是一个很好的练习,以找出如何提取该信息-特别是因为我一直发现自己要手动进行处理。
这是我当前的代码
#!/usr/bin/env python3
# REQUIREMENTS
# python3
# BeautifulSoup (pip3 install beautifulsoup)
# python 3 certificates (Applications/Python 3.x/ Install Certificates.command) <-- this one took me forever to figure out!
import sys
if sys.version_info[0] < 3:
raise Exception("Use Python 3: python3 " + sys.argv[0])
from urllib.request import urlopen
from bs4 import BeautifulSoup
import re
#specify/get the url to scrape
#url ='https://chromereleases.googleblog.com/2020/02/stable-channel-update-for-desktop.html'
#url = 'https://source.android.com/security/bulletin/2020-02-01.html'
url = input("What is the URL? ") or 'https://chromereleases.googleblog.com/2020/02/stable-channel-update-for-desktop.html'
print("Checking URL: " + url)
# CVE regular expression
cve_pattern = 'CVE-\d{4}-\d{4,7}'
# query the website and return the html
page = urlopen(url).read()
# parse the html returned using beautiful soup
soup = BeautifulSoup(page, 'html.parser')
count = 0
############################################################
# ANDROID === search for CVE references within <td> tags ===
# find all <td> tags
all_tds = soup.find_all("td")
#print all_tds
for td in all_tds:
if "cve" in td.text.lower():
print(td.text)
############################################################
# CHROME === search for CVE reference within <span> tags ===
# find all <span> tags
all_spans = soup.find_all("span")
for span in all_spans:
# this code returns results in triplicate
for i in re.finditer(cve_pattern, span.text):
count += 1
print(count, i.group())
# this code works, but only returns the first match
# match = re.search(cve_pattern,span.text)
# if match:
# print(match.group(0))
我为Android URL所做的工作正常-我遇到的问题是Chrome URL。他们在标签内包含CVE信息,我正在尝试利用正则表达式将其排除。
使用重新定义方法,最终得到的结果一式三份。使用re.search方法,它错过了CVE-2019-19925-他们在同一行上列出了两个CVE。
您能否提供有关最佳方法的任何建议?
谢谢!
我终于自己解决了。无需BeautifulSoup;一切现在都是RegEx。为了解决我之前看到的重复/重复结果,我将re.findall列表结果转换为字典(保留唯一值的顺序),然后又返回到列表。
import sys
if sys.version_info[0] < 3:
raise Exception("Use Python 3: python3 " + sys.argv[0])
import requests
import re
# Specify/get the url to scrape (included a default for easier testing)
### there is no input validation taking place here ###
url = input("What is the URL? ") #or 'https://chromereleases.googleblog.com/2020/02/stable-channel-update-for-desktop.html'
print()
# CVE regular expression
cve_pattern = r'CVE-\d{4}-\d{4,7}'
# query the website and return the html
page = requests.get(url)
# initialize count to 0
count = 0
#search for CVE references using RegEx
cves = re.findall(cve_pattern, page.text)
# after several days of fiddling, I was still getting double and sometimes triple results on certain pages. This next line
# converts the list of objects returned from re.findall to a dictionary (which retains order) to get unique values, then back to a list.
# (thanks to https://stackoverflow.com/a/48028065/9205677)
# I found order to be important sometimes, as the most severely rated CVEs are often listed first on the page
cves = list(dict.fromkeys(cves))
# print the results to the screen
for cve in cves:
print(cve)
count += 1
print()
print(str(count) + " CVEs found at " + url)
print()