我看过几篇关于在 Databricks 中使用 Selenium 的帖子，使用
%sh以下链接显示有同样问题但没有明确答案的人：
https://forums.databricks.com/questions/19376/if-my-notebook-downloads-a-file-from-a-website-by.html
当我使用 Selenium Python 进行 Web 自动化时，有没有办法确定文件在 Azure Databricks 中的下载位置？
还有一些人在努力让 Selenium 正常运行： https://forums.databricks.com/questions/14814/selenium-in-databricks.html
是否有在 Databricks 上使用 Selenium 和管理下载文件的明确指南？如何让 Chrome 浏览器和 ChromeDriver 版本自动保持同步？
这里是安装 Selenium、Chrome 和 ChromeDriver 的指南。这也会在通过 Selenium 下载文件后将文件移动到已安装的存储中。每个数字都应该在自己的单元格中。
%pip install selenium
import pickle as pkl
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
/tmp/version\$%sh
version=`curl -sS https://chromedriver.storage.googleapis.com/LATEST_RELEASE`
wget -N https://chromedriver.storage.googleapis.com/\${version}/chromedriver_linux64.zip -O /tmp/chromedriver_linux64.zip
/tmp/%sh
unzip /tmp/chromedriver_linux64.zip -d /tmp/chromedriver/
%sh
sudo curl -sS -o - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add
sudo echo "deb https://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google-chrome.list
sudo apt-get -y update
sudo apt-get -y install google-chrome-stable
** 步骤 3 - 5 可以合并为一个命令。您还可以使用以下命令创建 shell 脚本并将其用作 init 文件来配置集群，这在使用使用临时集群的作业集群时特别有用，因为 init 脚本适用于所有工作节点而不仅仅是驱动程序节点。这还会安装 Selenium，允许您跳过步骤 1。只需将一个单元格粘贴到新笔记本中，运行，然后将初始化脚本指向
dbfs:/init/init_selenium.sh%sh
# dbfs:/init/init_selenium.sh
cat > /dbfs/init/init_selenium.sh <<EOF
#!/bin/sh
echo Install Chrome and Chrome driver
version=`curl -sS https://chromedriver.storage.googleapis.com/LATEST_RELEASE`
wget -N https://chromedriver.storage.googleapis.com/\${version}/chromedriver_linux64.zip -O /tmp/chromedriver_linux64.zip
unzip /tmp/chromedriver_linux64.zip -d /tmp/chromedriver/
sudo curl -sS -o - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add
sudo echo "deb https://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google-chrome.list
sudo apt-get -y update
sudo apt-get -y install google-chrome-stable
pip install selenium
EOF
cat /dbfs/init/init_selenium.sh
service_principal_id = "YOUR_SP_ID"
service_principle_key = "YOUR_SP_KEY"
tenant_id = "YOUR_TENANT_ID"
directory = "https://login.microsoftonline.com/" + tenant_id + "/oauth2/token"
configs = {"fs.azure.account.auth.type": "OAuth",
"fs.azure.account.oauth.provider.type": "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider",
"fs.azure.account.oauth2.client.id": service_principal_id,
"fs.azure.account.oauth2.client.secret": service_principle_key,
"fs.azure.account.oauth2.client.endpoint": directory,
"fs.azure.createRemoteFileSystemDuringInitialization": "true"}
mount_point = "/mnt/container-data/"
mount_point_main = "/dbfs/mnt/container-data/"
container = "container-data"
storage_account = "adlsgen2"
storage = "abfss://"+ container +"@"+ storage_account + ".dfs.core.windows.net"
utils_folder = mount_point + "utils/selenium/"
raw_folder = mount_point + "raw/"
if not any(mount_point in mount_info for mount_info in dbutils.fs.mounts()):
dbutils.fs.mount(
source = storage,
mount_point = mount_point,
extra_configs = configs)
print(mount_point + " has been mounted.")
else:
print(mount_point + " was already mounted.")
print(f"Utils folder: {utils_folder}")
print(f"Raw folder: {raw_folder}")
utilsmnt/container-data/utils/seleniumdef init_chrome_browser(download_path, chrome_driver_path, cookies_path, url):
"""
Instatiates a Chrome browser.
Parameters
----------
download_path : str
The download path to place files downloaded from this browser session.
chrome_driver_path : str
The path of the chrome driver executable binary (.exe file).
cookies_path : str
The path of the cookie file to load in (.pkl file).
url : str
The URL address of the page to initially load.
Returns
-------
Browser
Returns the instantiated browser object.
"""
options = Options()
prefs = {'download.default_directory' : download_path}
options.add_experimental_option('prefs', prefs)
options.add_argument('--no-sandbox')
options.add_argument('--headless')
options.add_argument('--disable-dev-shm-usage')
options.add_argument('--start-maximized')
options.add_argument('window-size=2560,1440')
print(f"{datetime.now()} Launching Chrome...")
browser = webdriver.Chrome(service=Service(chrome_driver_path), options=options)
print(f"{datetime.now()} Chrome launched.")
browser.get(url)
print(f"{datetime.now()} Loading cookies...")
cookies = pkl.load(open(cookies_path, "rb"))
for cookie in cookies:
browser.add_cookie(cookie)
browser.get(url)
print(f"{datetime.now()} Cookies loaded.")
print(f"{datetime.now()} Browser ready to use.")
return browser
/tmp/downloads/dbfs/dbfs/mnt/...browser = init_chrome_browser(
download_path="/tmp/downloads",
chrome_driver_path="/tmp/chromedriver/chromedriver",
cookies_path="/dbfs"+ utils_folder + "cookies.pkl",
url="YOUR_URL"
)
进行导航并下载所需的任何内容。
可选：检查您的下载位置。在此示例中，我下载了一个 CSV 文件，并将搜索下载的文件夹，直到找到该文件格式。
import os
import os.path
for root, directories, filenames in os.walk('/tmp'):
print(root)
if any(".csv" in s for s in filenames):
print(filenames)
break
/mnt/container-data/raw/file:dbutils.fs.cp("file:/tmp/downloads/file1.csv", f"{raw_folder}file2.csv')
我一直在使用第一个答案中的指南在 Databricks 中安装 Selenium，第三步对我不起作用，因为“${version}”之前有“\”符号。
%sh
version=`curl -sS https://chromedriver.storage.googleapis.com/LATEST_RELEASE`
wget -N https://chromedriver.storage.googleapis.com/${version}/chromedriver_linux64.zip -O /tmp/chromedriver_linux64.zip
此外，在使用 ChromeDriverManager 时更改 init_chrome_browser 函数并没有给我带来错误。
options = webdriver.ChromeOptions()
prefs = {'download.default_directory' : download_path}
options.add_experimental_option('prefs', prefs)
options.add_argument('--no-sandbox')
options.add_argument('--headless')
options.add_argument('--disable-dev-shm-usage')
options.add_argument('--start-maximized')
options.add_argument('window-size=2560,1440')
print(f"{datetime.now()} Launching Chrome...")
browser = webdriver.Chrome(service=ChromeService(ChromeDriverManager().install()), options=options)
我使用了@kindofhungry提供的答案以及此资源...
https://github.com/Pratyusha23/shell_script_webscraping/blob/main/google_chrome_version_matching.py
...最终让 Selenium 在我的 databricks 笔记本中工作。我必须调整一些东西才能让它为我工作，所以希望其他人发现这有帮助。
确保导入服务
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
在第 3 步我必须使用此代码
%sh
version=`curl -sS https://chromedriver.storage.googleapis.com/LATEST_RELEASE`
wget -N https://chromedriver.storage.googleapis.com/${version}/chromedriver_linux64.zip -O /tmp/chromedriver_linux64.zip
对于第 4 步，我必须这样做
%sh
sudo mkdir -p /tmp/chromedriver
sudo unzip -o /tmp/chromedriver_linux64.zip -d /tmp/chromedriver
对于第 5 步，我必须手动回归 Chrome 版本以匹配驱动程序
%sh
sudo apt-get update
wget https://dl.google.com/linux/chrome/deb/pool/main/g/google-chrome-stable/google-chrome-stable_114.0.5735.198-1_amd64.deb
sudo apt-get -y update
sudo apt -y install ./google-chrome*.deb