Issue
I try to take price from website, but there is a problem, because there are products with and without discounts and prices for these discount and non discount are located in different "enter image description here span". My code take only first price in website (btw website https://ua.iherb.com/new-products?p=1)
this is part of code
def get_page_data(html): soup = BeautifulSoup(html, 'lxml')
lis = soup.find_all('div', class_="product-cell-container col-xs-12 col-sm-12 col-md-8 col-lg-6")
for li in lis:
try:
name = li.find('div', class_='absolute-link-wrapper').find('a').get('title')
except:
name = ''
try:
url = li.find('div', class_='absolute-link-wrapper').find('a').get('href')
except:
url = ''
prices = li.find_all('div', attrs={"class":"product-price-top"})
main_list=[]
for price in prices:
try:
discount_price = price.find("span", class_='price discount-green').text.strip()
main_list.append(discount_price)
except AttributeError:
original_price = price.find("span", class_='price ').text.strip()
main_list.append(original_price)
print(main_list)
Solution
You can scrape the AJAX API accessed by Javascript to bring the information into page. Here is one way to do it:
import requests
import pandas as pd
from tqdm import tqdm
pd.set_option('display.max_columns', None)
pd.set_option('display.max_colwidth', None)
# pd.set_option('display.max_rows', None)
s = requests.Session()
big_df = pd.DataFrame()
for x in tqdm(range(1, 400, 50)):
df = pd.json_normalize(s.get(f'https://ua.iherb.com/catalog/iherblive?isAjax=true&index={x}&nop=50&_=1665000595915').json())
big_df = pd.concat([big_df, df], axis=0, ignore_index=True)
big_df.drop('Product.RatingStarsMap', axis=1, inplace=True)
big_df.to_csv('ukr_herbs.csv')
print(big_df)
Result in terminal:
100%
8/8 [00:04<00:00, 1.69it/s]
Country CountryCode Index Product.Id Product.Name Product.ProductUrl Product.ProductImage Product.ProductImageRetina Product.PartNumber Product.ListPrice Product.DiscountPrice Product.OutOfStock Product.StockStatus Product.HidePrice Product.Discontinued Product.NotAvailable Product.ShowDiscount Product.DiscountType Product.IsInCartDiscount Product.Rating Product.RatingCount Product.RatingText Product.RatingURL Product.ReviewURL Product.IsShippingSaver Product.IsFeaturedBrand Product.IsAutoship Product.ShowGroupMessage Product.IsProductCompared Product.IsSeasonallyUnavailable Product.DiscountPercentage Product.PrimaryImageIndex Product.CartProductsInfo.lineItems Product.HasRating Product.HasTag Product.SalesDiscountPercentage Product.IsAutoApplyPromo Product.IsNew Product.IsTrial Product.SpecialDealInfo.PercentageClaimed Product.SpecialDealInfo.IsCompletelyClaimed Product.SpecialDealInfo.CountPerCustomer
0 Mexico mx 1 18492 Thorne Research, Bio-Gest, 180 Capsules https://www.iherb.com/pr/thorne-research-bio-gest-180-capsules/18492 https://cloudinary.images-iherb.com/image/upload/f_auto,q_auto:eco/images/thr/thr40502/k/13.jpg https://cloudinary.images-iherb.com/image/upload/f_auto,q_auto:eco/images/thr/thr40502/r/13.jpg THR-40502 $44.00 $44.00 False 0 False False False False 7 False 4.8 1284 4.8 of 5 based on 1284 https://www.iherb.com/pr/thorne-research-bio-gest-180-capsules/18492#product-detail-reviews https://www.iherb.com/r/thorne-research-bio-gest-180-capsules/18492 False False False False False False 0 13 [{'productId': 18492, 'productName': 'Thorne Research, Bio-Gest, 180 Capsules', 'iURLSmall': 'https://cloudinary.images-iherb.com/image/upload/f_auto,q_auto:eco/images/thr/thr40502/s/13.jpg', 'iURLMedium': 'https://cloudinary.images-iherb.com/image/upload/f_auto,q_auto:eco/images/thr/thr40502/m/13.jpg', 'listPrice': ' 44.00′,′𝑑𝑖𝑠𝑐𝑜𝑢𝑛𝑡𝑃𝑟𝑖𝑐𝑒′:′ 44.00', 'PartNumber': 'THR-40502'}] True False 0.0 False False False NaN NaN NaN
1 Israel il 2 103768 LoveBug Probiotics, Baby Probiotics, 6-12 Months, 4 Billion CFU, 30 Single Serve Stick Packs https://www.iherb.com/pr/lovebug-probiotics-baby-probiotics-6-12-months-4-billion-cfu-30-single-serve-stick-packs/103768 https://cloudinary.images-iherb.com/image/upload/f_auto,q_auto:eco/images/lvb/lvb00033/k/28.jpg https://cloudinary.images-iherb.com/image/upload/f_auto,q_auto:eco/images/lvb/lvb00033/r/28.jpg LVB-00033 $22.49 $22.49 False 0 False False False False 0 False 4.8 125 4.8 of 5 based on 125 https://www.iherb.com/pr/lovebug-probiotics-baby-probiotics-6-12-months-4-billion-cfu-30-single-serve-stick-packs/103768#product-detail-reviews https://www.iherb.com/r/lovebug-probiotics-baby-probiotics-6-12-months-4-billion-cfu-30-single-serve-stick-packs/103768 False False False False False False 0 28 [{'productId': 103768, 'productName': 'LoveBug Probiotics, Baby Probiotics, 6-12 Months, 4 Billion CFU, 30 Single Serve Stick Packs', 'iURLSmall': 'https://cloudinary.images-iherb.com/image/upload/f_auto,q_auto:eco/images/lvb/lvb00033/s/28.jpg', 'iURLMedium': 'https://cloudinary.images-iherb.com/image/upload/f_auto,q_auto:eco/images/lvb/lvb00033/m/28.jpg', 'listPrice': ' 22.49′,′𝑑𝑖𝑠𝑐𝑜𝑢𝑛𝑡𝑃𝑟𝑖𝑐𝑒′:′ 22.49', 'PartNumber': 'LVB-00033'}] True False 0.0 False False False NaN NaN NaN
2 United States us 3 8647 Now Foods, Modified Citrus Pectin, 800 mg, 180 Veg Capsules https://www.iherb.com/pr/now-foods-modified-citrus-pectin-800-mg-180-veg-capsules/8647 https://cloudinary.images-iherb.com/image/upload/f_auto,q_auto:eco/images/now/now03075/k/7.jpg https://cloudinary.images-iherb.com/image/upload/f_auto,q_auto:eco/images/now/now03075/r/7.jpg NOW-03075 $55.86 $55.86 False 0 False False False False 0 False 4.7 137 4.7 of 5 based on 137 https://www.iherb.com/pr/now-foods-modified-citrus-pectin-800-mg-180-veg-capsules/8647#product-detail-reviews https://www.iherb.com/r/now-foods-modified-citrus-pectin-800-mg-180-veg-capsules/8647 False False False False False False 0 7 [{'productId': 8647, 'productName': 'Now Foods, Modified Citrus Pectin, 800 mg, 180 Veg Capsules', 'iURLSmall': 'https://cloudinary.images-iherb.com/image/upload/f_auto,q_auto:eco/images/now/now03075/s/7.jpg', 'iURLMedium': 'https://cloudinary.images-iherb.com/image/upload/f_auto,q_auto:eco/images/now/now03075/m/7.jpg', 'listPrice': ' 55.86′,′𝑑𝑖𝑠𝑐𝑜𝑢𝑛𝑡𝑃𝑟𝑖𝑐𝑒′:′ 55.86', 'PartNumber': 'NOW-03075'}] True False 0.0 False False False NaN NaN NaN
3 United States us 4 101229 Blithe, Pressed Serum, Tundra Chaga, 1.68 fl oz (50 ml) https://www.iherb.com/pr/blithe-pressed-serum-tundra-chaga-1-68-fl-oz-50-ml/101229 https://cloudinary.images-iherb.com/image/upload/f_auto,q_auto:eco/images/blt/blt66010/k/2.jpg https://cloudinary.images-iherb.com/image/upload/f_auto,q_auto:eco/images/blt/blt66010/r/2.jpg BLT-66010 $38.00 $34.20 False 0 False False False True 1 False 4.6 9667 4.6 of 5 based on 9667 https://www.iherb.com/pr/blithe-pressed-serum-tundra-chaga-1-68-fl-oz-50-ml/101229#product-detail-reviews https://www.iherb.com/r/blithe-pressed-serum-tundra-chaga-1-68-fl-oz-50-ml/101229 False False False False False False 10 2 [{'productId': 101229, 'productName': 'Blithe, Pressed Serum, Tundra Chaga, 1.68 fl oz (50 ml)', 'iURLSmall': 'https://cloudinary.images-iherb.com/image/upload/f_auto,q_auto:eco/images/blt/blt66010/s/2.jpg', 'iURLMedium': 'https://cloudinary.images-iherb.com/image/upload/f_auto,q_auto:eco/images/blt/blt66010/m/2.jpg', 'listPrice': ' 38.00′,′𝑑𝑖𝑠𝑐𝑜𝑢𝑛𝑡𝑃𝑟𝑖𝑐𝑒′:′ 34.20', 'PartNumber': 'BLT-66010'}] True False 10.0 False False False 0.0 False 0.0
4 Korea, Republic of kr 5 62027 California Gold Nutrition, Silymarin Complex, Milk Thistle Extract Plus Dandelion, Artichoke, Curcumin C3 Complex®, Ginger, and BioPerine®, 120 Veggie Capsules https://www.iherb.com/pr/california-gold-nutrition-silymarin-complex-milk-thistle-extract-plus-dandelion-artichoke-curcumin-c3-complex-ginger-and-bioperine-120-veggie-capsules/62027 https://cloudinary.images-iherb.com/image/upload/f_auto,q_auto:eco/images/cgn/cgn00956/k/114.jpg https://cloudinary.images-iherb.com/image/upload/f_auto,q_auto:eco/images/cgn/cgn00956/r/114.jpg CGN-00956 $14.00 $14.00 False 0 False False False False 0 False 4.8 35521 4.8 of 5 based on 35521 https://www.iherb.com/pr/california-gold-nutrition-silymarin-complex-milk-thistle-extract-plus-dandelion-artichoke-curcumin-c3-complex-ginger-and-bioperine-120-veggie-capsules/62027#product-detail-reviews https://www.iherb.com/r/california-gold-nutrition-silymarin-complex-milk-thistle-extract-plus-dandelion-artichoke-curcumin-c3-complex-ginger-and-bioperine-120-veggie-capsules/62027 False False False False False False 0 114 [{'productId': 62027, 'productName': 'California Gold Nutrition, Silymarin Complex, Milk Thistle Extract Plus Dandelion, Artichoke, Curcumin C3 Complex®, Ginger, and BioPerine®, 120 Veggie Capsules', 'iURLSmall': 'https://cloudinary.images-iherb.com/image/upload/f_auto,q_auto:eco/images/cgn/cgn00956/s/114.jpg', 'iURLMedium': 'https://cloudinary.images-iherb.com/image/upload/f_auto,q_auto:eco/images/cgn/cgn00956/m/114.jpg', 'listPrice': ' 14.00′,′𝑑𝑖𝑠𝑐𝑜𝑢𝑛𝑡𝑃𝑟𝑖𝑐𝑒′:′ 14.00', 'PartNumber': 'CGN-00956'}] True False 0.0 False False False NaN NaN NaN
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
388 Japan jp 396 7344 Now Foods, B-12, 2,000 mcg, 100 Lozenges https://www.iherb.com/pr/now-foods-b-12-2-000-mcg-100-lozenges/7344 https://cloudinary.images-iherb.com/image/upload/f_auto,q_auto:eco/images/now/now00459/k/14.jpg https://cloudinary.images-iherb.com/image/upload/f_auto,q_auto:eco/images/now/now00459/r/14.jpg NOW-00459 $9.32 $9.32 False 0 False False False False 0 False 4.8 1198 4.8 of 5 based on 1198 https://www.iherb.com/pr/now-foods-b-12-2-000-mcg-100-lozenges/7344#product-detail-reviews https://www.iherb.com/r/now-foods-b-12-2-000-mcg-100-lozenges/7344 False False False False False False 0 14 [{'productId': 7344, 'productName': 'Now Foods, B-12, 2,000 mcg, 100 Lozenges', 'iURLSmall': 'https://cloudinary.images-iherb.com/image/upload/f_auto,q_auto:eco/images/now/now00459/s/14.jpg', 'iURLMedium': 'https://cloudinary.images-iherb.com/image/upload/f_auto,q_auto:eco/images/now/now00459/m/14.jpg', 'listPrice': ' 9.32′,′𝑑𝑖𝑠𝑐𝑜𝑢𝑛𝑡𝑃𝑟𝑖𝑐𝑒′:′ 9.32', 'PartNumber': 'NOW-00459'}] True False 0.0 False False False NaN NaN NaN
389 Hong Kong hk 397 106622 NutraMedix, Vitamins D3 & K2, 90 Capsules https://www.iherb.com/pr/nutramedix-vitamins-d3-k2-90-capsules/106622 https://cloudinary.images-iherb.com/image/upload/f_auto,q_auto:eco/images/ndx/ndx09348/k/8.jpg https://cloudinary.images-iherb.com/image/upload/f_auto,q_auto:eco/images/ndx/ndx09348/r/8.jpg NDX-09348 $29.95 $29.95 False 0 False False False False 0 False 4.7 46 4.7 of 5 based on 46 https://www.iherb.com/pr/nutramedix-vitamins-d3-k2-90-capsules/106622#product-detail-reviews https://www.iherb.com/r/nutramedix-vitamins-d3-k2-90-capsules/106622 False False False False False False 0 8 [{'productId': 106622, 'productName': 'NutraMedix, Vitamins D3 & K2, 90 Capsules', 'iURLSmall': 'https://cloudinary.images-iherb.com/image/upload/f_auto,q_auto:eco/images/ndx/ndx09348/s/8.jpg', 'iURLMedium': 'https://cloudinary.images-iherb.com/image/upload/f_auto,q_auto:eco/images/ndx/ndx09348/m/8.jpg', 'listPrice': ' 29.95′,′𝑑𝑖𝑠𝑐𝑜𝑢𝑛𝑡𝑃𝑟𝑖𝑐𝑒′:′ 29.95', 'PartNumber': 'NDX-09348'}] True False 0.0 False False False NaN NaN NaN
390 Australia au 398 2715 Natural Factors, Coenzyme Q10, 100 mg, 60 Softgels https://www.iherb.com/pr/natural-factors-coenzyme-q10-100-mg-60-softgels/2715 https://cloudinary.images-iherb.com/image/upload/f_auto,q_auto:eco/images/nfs/nfs02071/k/24.jpg https://cloudinary.images-iherb.com/image/upload/f_auto,q_auto:eco/images/nfs/nfs02071/r/24.jpg NFS-02071 $16.07 $16.07 False 0 False False False False 0 False 4.7 7700 4.7 of 5 based on 7700 https://www.iherb.com/pr/natural-factors-coenzyme-q10-100-mg-60-softgels/2715#product-detail-reviews https://www.iherb.com/r/natural-factors-coenzyme-q10-100-mg-60-softgels/2715 False False False False False False 0 24 [{'productId': 2715, 'productName': 'Natural Factors, Coenzyme Q10, 100 mg, 60 Softgels', 'iURLSmall': 'https://cloudinary.images-iherb.com/image/upload/f_auto,q_auto:eco/images/nfs/nfs02071/s/24.jpg', 'iURLMedium': 'https://cloudinary.images-iherb.com/image/upload/f_auto,q_auto:eco/images/nfs/nfs02071/m/24.jpg', 'listPrice': ' 16.07′,′𝑑𝑖𝑠𝑐𝑜𝑢𝑛𝑡𝑃𝑟𝑖𝑐𝑒′:′ 16.07', 'PartNumber': 'NFS-02071'}] True False 0.0 False False False NaN NaN NaN
391 Philippines ph 399 46284 Life Extension, Vitamin D3, 25 mcg (1,000 IU), 250 Softgels https://www.iherb.com/pr/life-extension-vitamin-d3-25-mcg-1-000-iu-250-softgels/46284 https://cloudinary.images-iherb.com/image/upload/f_auto,q_auto:eco/images/lex/lex17512/k/179.jpg https://cloudinary.images-iherb.com/image/upload/f_auto,q_auto:eco/images/lex/lex17512/r/179.jpg LEX-17512 $9.75 $9.75 False 0 False False False False 7 False 4.8 2698 4.8 of 5 based on 2698 https://www.iherb.com/pr/life-extension-vitamin-d3-25-mcg-1-000-iu-250-softgels/46284#product-detail-reviews https://www.iherb.com/r/life-extension-vitamin-d3-25-mcg-1-000-iu-250-softgels/46284 False False False False False False 0 179 [{'productId': 46284, 'productName': 'Life Extension, Vitamin D3, 25 mcg (1,000 IU), 250 Softgels', 'iURLSmall': 'https://cloudinary.images-iherb.com/image/upload/f_auto,q_auto:eco/images/lex/lex17512/s/179.jpg', 'iURLMedium': 'https://cloudinary.images-iherb.com/image/upload/f_auto,q_auto:eco/images/lex/lex17512/m/179.jpg', 'listPrice': ' 9.75′,′𝑑𝑖𝑠𝑐𝑜𝑢𝑛𝑡𝑃𝑟𝑖𝑐𝑒′:′ 9.75', 'PartNumber': 'LEX-17512'}] True False 0.0 False False False NaN NaN NaN
392 Australia au 400 104625 I Dew Care, Plush Party, Buttery Vitamin C Lip Mask, 0.42 oz (12 g) https://www.iherb.com/pr/i-dew-care-plush-party-buttery-vitamin-c-lip-mask-0-42-oz-12-g/104625 https://cloudinary.images-iherb.com/image/upload/f_auto,q_auto:eco/images/idc/idc71810/k/13.jpg https://cloudinary.images-iherb.com/image/upload/f_auto,q_auto:eco/images/idc/idc71810/r/13.jpg IDC-71810 $16.00 $16.00 False 0 False False False False 0 False 4.6 73 4.6 of 5 based on 73 https://www.iherb.com/pr/i-dew-care-plush-party-buttery-vitamin-c-lip-mask-0-42-oz-12-g/104625#product-detail-reviews https://www.iherb.com/r/i-dew-care-plush-party-buttery-vitamin-c-lip-mask-0-42-oz-12-g/104625 False False False False False False 0 13 [{'productId': 104625, 'productName': 'I Dew Care, Plush Party, Buttery Vitamin C Lip Mask, 0.42 oz (12 g)', 'iURLSmall': 'https://cloudinary.images-iherb.com/image/upload/f_auto,q_auto:eco/images/idc/idc71810/s/13.jpg', 'iURLMedium': 'https://cloudinary.images-iherb.com/image/upload/f_auto,q_auto:eco/images/idc/idc71810/m/13.jpg', 'listPrice': ' 16.00′,′𝑑𝑖𝑠𝑐𝑜𝑢𝑛𝑡𝑃𝑟𝑖𝑐𝑒′:′ 16.00', 'PartNumber': 'IDC-71810'}] True False 0.0 False False False NaN NaN NaN
393 rows × 42 columns
Data is also saved to a csv file. You probably have some duplicates (I didn't match the AJAX numbers exactly), you can filter them out.
See relevant documentation for the libraries used:
Pandas: https://pandas.pydata.org/docs/
TQDM: https://pypi.org/project/tqdm/
And Requests: https://requests.readthedocs.io/en/latest/
Answered By - Barry the Platipus
0 comments:
Post a Comment
Note: Only a member of this blog may post a comment.