如何通过Scrapy获取城市名称

Question

我正在尝试使用以下网站的 scrapy 从每个电缆入口中抓取城市名称：https://www.submarinecablemap.com/

我的目标是列出所有作为电缆出口点的城市。 我也尝试过漂亮的汤。 我如何选择每条电缆，转到“href”并收集列表中列出的所有城市？

Answer 1

这个怎么样？需要错误检查，但应该可以完成工作。

import json
import time
import requests

response = requests.get("https://www.submarinecablemap.com/api/v3/cable/cable-geo.json")
cables = response.json()

ID = [feature["properties"]["id"] for feature in cables["features"]]
ID = sorted(ID)

CITIES = {}

for id in ID:
    print(id)
    response = requests.get(f"https://www.submarinecablemap.com/api/v3/cable/{id}.json")
    cable = response.json()

    cities = [landing["name"] for landing in cable["landing_points"]]

    CITIES[id] = cities

    # Being polite.
    time.sleep(5)

with open("cable-cities.json", "wt") as file:
    json.dump(CITIES, file, indent=2)

以下是两条电缆的数据。数据排列为 JSON 对象，其中电缆段为键，值为着陆点位置列表。

{
  "2africa": [
    "Luanda, Angola",
    "Manama, Bahrain",
    "Moroni, Comoros",
    "Muanda, Congo, Dem. Rep.",
    "Pointe-Noire, Congo, Rep.",
    "Abidjan, C\u00f4te d'Ivoire",
    "Djibouti City, Djibouti",
    "Port Said, Egypt",
    "Ras Ghareb, Egypt",
    "Suez, Egypt",
    "Zafarana, Egypt",
    "Marseille, France",
    "Libreville, Gabon",
    "Accra, Ghana",
    "Tympaki, Greece",
    "Mumbai, India",
    "Al Faw, Iraq",
    "Genoa, Italy",
    "Mombasa, Kenya",
    "Mtwapa, Kenya",
    "Kuwait City, Kuwait",
    "Mahajanga, Madagascar",
    "Maputo, Mozambique",
    "Nacala, Mozambique",
    "Kwa Ibo, Nigeria",
    "Lagos, Nigeria",
    "Barka, Oman",
    "Salalah, Oman",
    "Karachi, Pakistan",
    "Carcavelos, Portugal",
    "Doha, Qatar",
    "Al Khobar, Saudi Arabia",
    "Duba, Saudi Arabia",
    "Jeddah, Saudi Arabia",
    "Yanbu, Saudi Arabia",
    "Dakar, Senegal",
    "Carana, Seychelles",
    "Berbera, Somalia",
    "Mogadishu, Somalia",
    "Amanzimtoti, South Africa",
    "Gqeberha, South Africa",
    "Melkbosstrand, South Africa",
    "Yzerfontein, South Africa",
    "Barcelona, Spain",
    "Gran Canaria, Canary Islands, Spain",
    "Port Sudan, Sudan",
    "Dar Es Salaam, Tanzania",
    "Abu Dhabi, United Arab Emirates",
    "Kalba, United Arab Emirates",
    "Bude, United Kingdom"
  ],
  "acs-alaska-oregon-network-akorn": [
    "Anchorage, AK, United States",
    "Florence, OR, United States",
    "Homer, AK, United States",
    "Nikiski, AK, United States"
  ]
}