[FIXED] how to do a web scraping getting all the data

Issue

I was doing a web scrapping to get reviews from the play store in order to study NLP, but my script only brought the first reviews.

I noticed that when we clicked on "Show all reviews" the classes of the elements of the comments on the html page remain the same and the URL too, so theoretically all the data should come.

Can anyone help me?

import requests
from bs4 import BeautifulSoup

def scrape_playstore_simpler_reviews(url):
    reviews_list = []
    response = requests.get(url)

    if response.status_code == 200:
        soup = BeautifulSoup(response.content, 'html.parser')
        app_title1 = soup.find_all('h1', class_='Fd93Bb')

        for app_title in app_title1:
            app_title1 = app_title.find('span')


        reviews = soup.find_all('div', class_='EGFGHd')

        for review in reviews:
            user_name = review.find('div', class_='X5PpBb')
            comment_text = review.find('div', class_='h3YV2d')
            helpful_count = review.find('div', class_='AJTPZc')
            post_date = review.find('span', class_='bp9Aid')
            star_rating = review.find('div', class_='iXRFPc').attrs['aria-label']
            
            review_data = {
                'Título do aplicativo': app_title,
                'Nome do usuário': user_name,
                'Texto do comentário': comment_text,
                'Qtd apoio ao comentário': helpful_count,
                'Data da postagem': post_date,
                'Qtd estrelas': star_rating
            }

            reviews_list.append(review_data)
            print(review_data)
    else:
        print('Falha ao fazer requisição HTTP')

url = 'https://play.google.com/store/apps/details?id=ru.zengalt.simpler&hl=pt_BR'
reviews = scrape_playstore_simpler_reviews(url)

scrape_playstore_simpler_reviews(url)

I expected to see all the app rating comments I looked for

Solution

It seems that the issue with your web scraping script lies in how you're trying to extract the reviews from the Google Play Store page. The reason you're only getting the first reviews is that the page uses dynamic loading, and the reviews are loaded as you scroll down.

To scrape all the reviews, you'll need to simulate scrolling or use the Play Store API to get all the reviews. Directly scraping the page won't work since the initial response doesn't contain all the reviews.

Here's a general outline of how you can scrape all reviews using the Play Store API:

1. Use the Play Store API to fetch the reviews data.
2. Parse the JSON response to extract the necessary information.

Answered By - NOX