如何使用Selenium + Python加载动态内容

问题描述 投票:0回答:3

在Quora用户个人资料页面上,我想通过点击“更多”来显示每个用户的答案的整个文本,但我无法获得“更多”元素列表,而是我得到一个空列表,该怎么做?

这是我的代码:

more_buttons = browser.find_elements_by_name('more')
for more_button in more_buttons:
        more_button.click()

以下是用户个人资料的示例:https://www.quora.com/profile/Ashley-Isabella-3

python selenium selenium-webdriver web-scraping webdriverwait
3个回答
2
投票

那是因为你使用了错误的CSS Selector。试试:

more_buttons = browser.find_elements_by_class_name('ui_qtext_more_link')

通过在普通浏览器中右键单击元素并单击“检查”,可以找到正确的属性。


2
投票

在Quora用户配置文件页面上单击带有文本的链接(更多)并检索用户答案,您可以使用以下解决方案:

  • 代码块: from selenium import webdriver from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.common.by import By from selenium.webdriver.support import expected_conditions as EC driver=webdriver.Firefox(executable_path=r'C:\Utility\BrowserDrivers\geckodriver.exe') driver.get("https://www.quora.com/profile/Ashley-Isabella-3") more_buttons = WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "a.ui_qtext_more_link"))) for more_button in more_buttons: more_button.click() my_text_elements = WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "div.ui_qtext_expanded>span.ui_qtext_rendered_qtext>p>b"))) for my_text_element in my_text_elements: print(my_text_element.text)
  • 控制台输出: 11 mistakes you make when trying to lose belly fat You could be doing the completely wrong exercise. 1. You're not getting enough sleep 2. You’re doing the wrong type of workout 3. You’re consuming too much sugar 4. You're not eating enough protein 5. You’re feeling stressed or anxious 6. You’re expecting a quick fix 7. You’re not tracking your progress 8. You’re crash-dieting 9. You’re doing too much 10. You haven't got the right exercise balance 11. You’ve lost motivation 9 Ways To Burn Fat Fast Implement these 9 fat-burning tips that use exercise and diet and watch the body fat melt like the butter you're no longer using. 1. Stay Off The Scale 2. Reduce Your Calories Gradually 3. Vary Your Caloric Intake 4. Train With Weights 5. Do High-Intensity Intervals (HIIT) 6. Eat More Fat 7. Cut Carbs 8. Increase Your Protein 9. Eat 6 Smaller Meals Per Day, Not 2-3 Feasts Bonus: Researchers Explain 6 Reasons Why A Man Falls In Love 1. HE’S CAPTIVATED BY THE WHOLE PACKAGE 2. HE FEELS HE CAN MAKE HER HAPPY 3. WHEN SHE’S OPEN TO LOVE 4. THE WAY HE FEELS AROUND HER 5. SHE SAYS “YES” TO LIFE 6. HE FEELS PURPOSE AND PASSION Must Watch: Secret For You in Under 20 Minutes... These are the 7 weirdest things men find attractive in women Who knew anyone would ever find stubbornness attractive in a woman? 1. Weakness 2. Applying makeup 3. Stubbornness 4. Stretch marks 5. Wearing glasses 6. No make up 7. Clumsiness Secret For You in Under 20 Minutes... 5 surprising ways to get great skin There are so many bad ideas out there about how to achieve great skin. Here are five ways to get on the right track, regardless of 'the rules.' 1. You don't need to wash your face with hot water to get it clean. 2. Oil is good for your skin, not bad. 3. You don't need to scrub to exfoliate. 4. What matters most for healthy skin is not what you put on it, but what you eat. 5. Chocolate doesn't cause acne, but bread and pasta might Make Your Skin Naturally Glow 1. Skip the long, steamy showers and opt for shorter, cooler sprays. 2. Check the dryness of your skin by scratching a small area on your arm or leg with your fingernail. 3. Treat your neck and chest like an extension of your face. 4. Run a humidifier every night in the winter to moisturize the air in your bedroom. 12 easy tips for glowing skin 1: Discover these glowing skin home remedies: 2: Sweat it out for healthy skin 3: Practice Yoga for Face Glow 4: Know who you are! 5: Follow a natural diet for glowing skin 6: Indulge yourself…weekly 7: Make Sudarshan Kriya your beauty mantra 8: Meditate everyday to get Glowing Skin 9: Silence is truly golden 10: Save your mind at any cost 11: 18 till I …why not? 12: Flex those facial muscles 15 Reasons Yoga Is Better Than The Gym 1. Yoga benefits the mind, body and spirit. 2. Yoga benefits your full body—externally and internally. 3. Yoga teaches acceptance. 4. Yoga is about focusing on yourself. 5. Yoga will make you lean. 6. Yoga is more efficient. 7. You can do yoga anywhere. 8. Yoga is kinder to the body. 9. Yoga eases your aches and pains. 10. Yoga helps you breathe easier. 11. Yoga is calming. 12. Yoga reduces stress. 13. Everyone can practice yoga. 14. Yoga improves concentration. 15. Yogis rock. 9 Ways to Tell If You’re Exercising Enough The sign: You never feel sore or tired the day after a workout The sign: You're not losing any weight The sign: You barely break a sweat The sign: The last rep feels hard, but you could probably do another The sign: You can chat with your friend throughout your workout The sign: You're not getting any more flexible in your yoga classes The sign: You run several days a week but aren't getting faster The sign: You're making strides in boot camp but are always stiff and un-flexible The sign: You take the same class regularly but aren't getting better at the exercises How To Build Muscle Mass After 50 Exercise Plan Strength-Training Exercises Weight, Reps and Sets Aerobics Workouts Savannah Guthrie Says The Keto Diet Is Making Her Mind Sharp—And Her Body Sluggish She has zero desire to work out. How to Lose Weight in 10 Days at Home Flight of Stairs / Treadmill Lots of Water Fresh – Fruits and Vegetables – Low-Fat Foods Fish / Meat – Avoid Junk Food – Moderate Exercise - Drink lots of water - Day 1 Day 2 and 3 Day 4 and 5 Day 6 and 7 Day 8 and 9 Day 10 What are natural ways to make my face look younger? Smoothing Exfoliator Tightening Toner Plumping Moisturizing Facial Mask 10 Ways to Lose 25 Pounds Follow these steps and weight loss won’t seem impossible. 1. Do some detective work. 2. Get moving. 3. Plan ahead. 4. Fill up on fruits and veggies. 5. Don’t drink your calories. 6. Don’t do anything drastic. 7. Set smart calorie targets. 8. Keep track. 9. Do not make weight the only measure of success. 10. Just practice; don’t try to be perfect.

2
投票

实际上,您在以下链接之后(当您单击时),因此您可以收集链接,然后使用.get循环。使用return [...document.querySelectorAll ]语法

from selenium import webdriver

d = webdriver.Chrome()
d.get("https://www.quora.com/profile/Ashley-Isabella-3")
links = d.execute_script("return [...document.querySelectorAll('.ui_qtext_more_link')].map(link => link['href'])")
info = []
for link in links:
    d.get(link)
    pageInfo = d.execute_script("return [...document.querySelectorAll('.ui_qtext_para')].map(link => link.innerText)")
    info.append(pageInfo)
print(info)
© www.soinside.com 2019 - 2024. All rights reserved.