[FIXED] extracting amazon best seller book price-web scraping

Issue

I am extracting Amazon bestseller books data book name, author name, and price of the book. For this task, I am using the beautifulSoup and requests library. The URL is- https://www.amazon.in/gp/bestsellers/books/ [code snippet-price infoHTML tree structure](https://i.stack.imgur.com/7K0rN.png)

I am getting an empty list of price info. How to fix the issue?

I have checked one of the post in Stackoverflow it suggests the use of lxml. Does it apply to my case also?

Solution

Here is an example how you can get titles of the books + their price:

import requests
from bs4 import BeautifulSoup

headers = {
    "User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:121.0) Gecko/20100101 Firefox/121.0"
}

url = "https://www.amazon.in/gp/bestsellers/books/"

soup = BeautifulSoup(requests.get(url, headers=headers).content, "html.parser")

for book in soup.select("div#gridItemRoot"):
    title = book.select_one("a:not(:has(img))").text
    price = book.select_one(".a-color-price").text
    print(f"{price:<10} {title}")

Prints:

₹115.00    The Power of Your Subconscious Mind: Original Edition | Premium Paperback
₹649.00    Indian Polity for UPSC (English)|7th Edition|Civil Services Exam| State Administrative Exams
₹97.00     MINtile Sank Magic Practice Copybook, (4 Book + 10 Refill) Number Tracing Book for Preschoolers with Pen, Magic Calligraphy Copybook Set Practical Reusable Writing Tool Simple Hand Lettering
₹511.00    Atomic Habits
₹220.00    The Psychology Of Money
₹409.00    My First Library: Boxset of 10 Board Books for Kids
₹184.00    Don't Believe Everything You Think (English)
₹339.00    Ikigai
₹156.00    Do It Today: Overcome procrastination, improve productivity and achieve more meaningful things [Paperback] Foroux, Darius
₹411.00    Educart 21 Days Challenge with Class 10 Final Revision Book for CBSE 2024 Board Exam- Maths + Science + SST + English (2023-24)
₹87.00     My First Book of Patterns Pencil Control: Patterns Practice book for kids (Pattern Writing)
₹97.00     201 Brain Booster Activity Book - Fun Activities and Exercises For Children: Tracing & Pattern, Colors & Shapes, Maze
₹545.00    Why Bharat Matters
₹180.00    BRAHMASTRA Complete Maths Multicolored Formula Book Second Edition BILINGUAL by Aditya Ranjan Sir
₹193.00    My First Mythology Tale (Illustrated) (Set of 5 Books) - Mahabharata, Krishna, Hanuman, Ganesha, Ramayana - Story Book for Kids - English Short Stories with Colourful Pictures - Read Aloud to Infants, Toddlers
₹296.00    Blackbook of English Vocabulary
₹97.00     You Can
₹211.00    Moral Story Books for Kids (Illustrated) - English Short Stories with Colourful Pictures - Bedtime Children Story Book - 3 Years to 6 Years Old Children - Read Aloud to Infants, Toddlers (Set of 10 Books)
₹97.00     Meditations
₹399.00    NTA UGC NET /SET/JRF Paper 1, Teaching and Research Aptitude – 2023, Includes latest 2022 paper and 2600+ Practice Questions with Solutions | Includes NEP - 2020| 7th Edition - By Pearson
₹97.00     The Power of Your Subconscious Mind
₹426.00    Rich Dad Poor Dad: 25th Anniversary Edit
₹187.00    Animals Tales From Panchtantra: Timeless Stories for Children From Ancient India
₹262.00    Oxford Student Atlas for India, Fourth Edition - Useful for Competitive Exams
₹93.00     How to Win Friends and Influence People : Original Edition | Premium Paperback
₹339.00    Colouring Books Boxset: Pack of 12 Copy Colour Books For Children
₹183.00    The Alchemist
₹156.00    Grandma's Bag of Stories: Collection of 20+ Illustrated short stories, traditional Indian folk tales for all ages for children of all ages by Sudha Murty [Paperback] Sudha Murty
₹147.00    Shlokas and Mantras - Activity Book For Kids - Illustrated Book With Engaging Activities and Sticker Sheets
₹94.00     Brain Activity Book for Kids - 200+ Activities for Age 3+

Answered By - Andrej Kesely