[FIXED] Element search in selenium in a loop

Issue

I am trying to retrieve data from this website: https://my.supplychain.nhs.uk/Catalogue/search?LastCartId=&LastFavouriteId=&Query=surgical+mask. To find all masks on the page I am doing this for every search page in a loop:

search_result = WebDriverWait(driver, 10).until(
            EC.presence_of_element_located((By.ID, "search_results"))      
        )
masks_result_list = search_result.find_elements_by_class_name("product_listing ")

It works for the first page of search, however, when I go to the next page of search, it returns masks_result_list as empty. I have also noticed that search_result: <selenium.webdriver.firefox.webelement.FirefoxWebElement (session="0249d71e-6f51-6147-8305-d2a2b738c29f", element="05367ec5-99fb-a14d-b039-214b128ed73b")>is the same element for every page. I am not proficient in selenium, so I don't know what can I do next to retrieve the data from all the pages. Is there a problem with my code or did I misunderstand the structure of the website?

Solution

You did not provide the entire your code, but I guess you are waiting for

search_result = WebDriverWait(driver, 10).until(
            EC.presence_of_element_located((By.ID, "search_results"))      
        )

immediately after pressing the next button to go to the next results page so the search_results element of the previous page is still present or something like this.
I suggest you to wait for disappearing of one of the products on the previous (current) page with EC.invisibility_of_element_located method and only after that to

search_result = WebDriverWait(driver, 10).until(
            EC.presence_of_element_located((By.ID, "search_results"))      
        )

and then to

masks_result_list = search_result.find_elements_by_class_name("product_listing ")

for the products on the new page.

Answered By - Prophet