Issue
The following retailer product page has an infinite scroll which requests data from a URL I am unable to access:
https://www.aldoshoes.com/uk/en_UK/men/holiday-party-style
Once "View All" is clicked, Developer tools states a URL which fails to show any product data and gives an "unsupported brand" error:
I would like to know if it is possible to access all of the product data through Scrapy without the need for Selenium.
Thank you.
Solution
If you look at the HTTP request that is made when you click on "View All" with your browser developer tools, you will notice that the request headers contain some application specific headers:
x-aldo-region: uk
x-aldo-lang: en_UK
x-aldo-brand: aldoshoes
x-forwarded-akamai-edgescape: undefined
x-aldo-ssr-request-id:
x-aldo-api-version: 2
Try to include these headers in your request.
Answered By - Guillaume
0 comments:
Post a Comment
Note: Only a member of this blog may post a comment.