[FIXED] Scraping webpage with show more button

Issue

I want to scrape google scholar pages with 'show more' button. Using help from this platform for a previous question that I had asked, I wrote the following code so that the 'show more' button is clicked. However, I am still having a problem. For profiles with several 'show more' buttons, only the first one is getting clicked. I dont understand why this happens. I would appreciate any help.

from selenium import webdriver
import time
from bs4 import BeautifulSoup
import pandas as pd

options = webdriver.ChromeOptions()
options.add_argument('--ignore-certificate-errors')
options.add_argument('--incognito')
chrome_path = r"C:\Users\ish05\Desktop\python\chromedriver.exe"
driver = webdriver.Chrome(chrome_path)

driver.get("https://scholar.google.com/citations?user=cp-8uaAAAAAJ&hl=en")
time.sleep(3)
show_more = driver.find_elements_by_tag_name('button')
for x in range(len(show_more)):
    if show_more[x].is_displayed():
      driver.execute_script("arguments[0].click();", show_more[x])
      time.sleep(3)

Solution

The reason it runs one because it appears one on each page.

You need to use infinite loop and then search on the page if there then click else no more button break from the loop.

from selenium import webdriver
import time
chrome_path = r"C:\Users\ish05\Desktop\python\chromedriver.exe"
driver = webdriver.Chrome(chrome_path)

driver.get("https://scholar.google.com/citations?user=cp-8uaAAAAAJ&hl=en")
time.sleep(3)
while True:
    try:       
        show_more = driver.find_element_by_xpath("//button[.//span[text()='Show more'] and not(@disabled)]")
        driver.execute_script("arguments[0].click();", show_more)
        print("Show more button clicked")
        time.sleep(2)
    except:
        print("No more Show more button")
        break

you will see below output on console

Show more button clicked
Show more button clicked
Show more button clicked
Show more button clicked
Show more button clicked
No more Show more button

Answered By - KunduK