从另一个文件中读取Urls和Scrape数据- Bash。

Question

我想从URL.txt中获取URL，然后将它们追加到基础URL的末尾。https://www.mcdelivery.com.pk/pk/browse/menu.html 它存在于另一个文件menu.sh中。

Url.text文件中包含

?daypartId=1&catId=1
?daypartId=1&catId=2
?daypartId=1&catId=11
?daypartId=1&catId=10
?daypartId=1&catId=6
?daypartId=1&catId=4
?daypartId=1&catId=14
?daypartId=1&catId=5
?daypartId=1&catId=3
?daypartId=1&catId=8

我想添加网址，如 https://www.mcdelivery.com.pk/pk/browse/menu.html?daypartId=1&catId=11 基本网址+来自URL.txt文件的网址。

我想出了这段代码，但问题是我只得到了第一页的价格，而且它一直在重复同一页的数值，直到循环结束。

ARRAY=()
while read -r LINE
do
ARRAY+=("$LINE")
done < URL.txt
for LINE in "${ARRAY[@]}"
do   
echo $LINE
curl https://www.mcdelivery.com.pk/pk/browse/menu.html$LINE | grep -o '<span class="starting-price">.*</span>' | sed 's/<[^>]\+>//g' >> price.txt 
done

我得到的输出

Rs 398
Rs 487
Rs 841
Rs 752
Rs 398
Rs 398
Rs 487
Rs 841
Rs 752
....

我想从每个页面获取价格，并将其存储到price.txt中。

Answer 1

#!/bin/bash
curl -sL https://www.mcdelivery.com.pk/pk/browse/menu.html | grep -o '<li class="secondary-menu-item ">.*</li>' | sed 's/href=/\nhref=/g' | \
grep 'href=\"' | \
sed 's/.*href="//g;s/".*//g' > URL.txt
sed -i 's/amp;//' URL.txt

ARRAY=()
while read -r LINE
do
    ARRAY+=("$LINE")
done < URL.txt

for LINE in "${ARRAY[@]}"
do    
    echo $LINE
    curl https://www.mcdelivery.com.pk/pk/browse/menu.html"$LINE" | grep -o '<h5 class="product-title">.*</h5>' | sed 's/<[^>]\+>//g' >> name.txt
    curl https://www.mcdelivery.com.pk/pk/browse/menu.html"$LINE" | grep -o '<span class="starting-price">.*</span>' | sed 's/<[^>]\+>//g' >> price.txt 
done    

这是我问题的答案。谢谢大家的帮助!!!

Answer 2

请不要使用正则表达式来解析html。. 使用一个真正的html-arser web-scraper，如 xidel 来代替。事实上，根本不需要Bash脚本。xidel 就可以完成你想要的一切。

解析html的"★最新动态★。"-menu-item和string-join价格+产品名称。

$ xidel -s "https://www.mcdelivery.com.pk/pk/browse/menu.html?daypartId=1&catId=1" -e '
  //div[ends-with(@class,"panel-product")]/join(
    (.//span[@class="starting-price"],.//h5),
    " - "
  )
'
Rs 288 - Cappuccino with Milk Chocolate Cookie
Rs 288 - Cappuccino with Double Chocolate Cookie
Rs 288 - Latte with Milk Chocolate Cookie
[...]
Rs 239 - Salted Caramel Shake

列出所有菜单项目和字符串连接网址+标题。

$ xidel -s https://www.mcdelivery.com.pk/pk/browse/menu.html -e '
  //ul[@class="secondary-menu"]//a/join((resolve-uri(@href),span)," - ")
'
https://www.mcdelivery.com.pk/pk/browse/menu.html?daypartId=1&catId=12 - Deals
https://www.mcdelivery.com.pk/pk/browse/menu.html?daypartId=1&catId=1 - ★What's New★
https://www.mcdelivery.com.pk/pk/browse/menu.html?daypartId=1&catId=2 - Ala carte & Value Meals
[...]
https://www.mcdelivery.com.pk/pk/browse/menu.html?daypartId=1&catId=8 - Snack Time

对于每一个菜单项，string-join url + title，打开url解析html和string-join price + product-name。

$ xidel -s https://www.mcdelivery.com.pk/pk/browse/menu.html -e '
  //ul[@class="secondary-menu"]//a/(
    join((resolve-uri(@href),span)," - "),
    doc(@href)//div[ends-with(@class,"panel-product")]/join(
      (.//span[@class="starting-price"],.//h5),
      " - "
    )
  )
'
https://www.mcdelivery.com.pk/pk/browse/menu.html?daypartId=1&catId=12 - Deals
Rs 487 - Grand Chicken Spicy with Drink
Rs 398 - Big Mac + Regular Drink
https://www.mcdelivery.com.pk/pk/browse/menu.html?daypartId=1&catId=1 - ★What's New★
Rs 288 - Cappuccino with Milk Chocolate Cookie
Rs 288 - Cappuccino with Double Chocolate Cookie
Rs 288 - Latte with Milk Chocolate Cookie
Rs 288 - Latte with Double Chocolate Cookie
Rs 159 - McFizz Guava
Rs 195 - Date Pie
Rs 416 - Spicy McCrispy Deluxe - Regular Meal
Rs 416 - McChicken - Regular Meal
Rs 239 - Curly Fries
Rs 239 - Salted Caramel Shake
https://www.mcdelivery.com.pk/pk/browse/menu.html?daypartId=1&catId=2 - Ala carte & Value Meals
Rs 257 - Chicken Burger with Cheese
Rs 265 - Value McArabia Chicken
Rs 265 - Mini McRoyale
[...]
https://www.mcdelivery.com.pk/pk/browse/menu.html?daypartId=1&catId=8 - Snack Time
Rs 301 - Spicy Chicken Burger
Rs 301 - 4pcs McNuggets
Rs 301 - Fries & Drink
Rs 150 - Apple Pie with Tea