[FIXED] stuck scraping the same 2nd page with infinite scroll

Issue

I'm trying to scrape game reviews from steam. when running the spider above, I get the first page with 10 reviews. then the second page with 10 reviews three times

class MySpider(scrapy.Spider):
    name = "MySpider"
    download_delay = 6
    page_number = 1
    start_urls = (
    'https://steamcommunity.com/app/1794680/reviews/', 
    )

    custom_settings = {
        'LOG_LEVEL': logging.WARNING,
        'LOG_ENABLED': False,
        'LOG_FILE': 'logging.txt',
        'LOG_FILE_APPEND': False,
        'REQUEST_FINGERPRINTER_IMPLEMENTATION': '2.7',
        'FEEDS': {"items.json": {"format": "json", 'overwrite': True},},
    }

    def parse(self, response):
        soup = BeautifulSoup(response.text, 'lxml')
        for review in soup.find_all('div', class_="apphub_UserReviewCardContent"):
           {...}
            
        if(self.page_number<4):
            self.page_number +=1
            yield scrapy.Request('https://steamcommunity.com/app/1794680/homecontent/?userreviewscursor=AoIIPwYYanu12fcD&userreviewsoffset={offset}&p={p}&workshopitemspage={p}&readytouseitemspage={p}&mtxitemspage={p}&itemspage={p}&screenshotspage={p}&videospage={p}&artpage={p}&allguidepage={p}&webguidepage={p}&integratedguidepage={p}&discussionspage={p}&numperpage=10&browsefilter=trendweek&browsefilter=trendweek&l=english&appHubSubSection=10&filterLanguage=default&searchText=&maxInappropriateScore=100'.format(offset=10*(self.page_number-1) ,p=self.page_number),method='GET', callback=self.parse)
            

json output

I took a few request when scrolling the reviews. I changed all values that looked like page number and replaced them with {p}, also I tried changing the 'userreviewsoffset' to fit the request format

i noticed that 'userreviewscursor' has a changing value every request but I don't know where it is from.

Solution

Your issue is with userreviewscursor=AoIIPwYYanu12fcD part of the url. That bit will change for every call, and you can find it in the HTML response under:

<input type="hidden" name="userreviewscursor" value="AoIIPwYYanLi8vYD">

Get that value and add it to the next call, and you're alright. (didn't want to babysit you and write the full code, but if needs be, let me know).

Answered By - Barry the Platipus