[FIXED] Webscraping specific fields in Python

Issue

How can I extract the company and its description from here?

From my yesterday's question I figured out how names are extracted but when I applied the same logic to extract their description it backfired.

request = requests.get("https://www.clstack.com", verify=False, headers=headers)
soup = bs4.BeautifulSoup(request.content, 'html.parser')
data = soup.find_all('td', {'class':'company'})

for i in data:
    print(i.find['tr'])

output

company|description

The desc is inside 'td' tag but when I call it from code, I don't get any output.

Solution

You'll notice that the <td class="company"> tags are followed by another <td> tag with the the description. So once you're iterating through the <td class="company"> element, simply use the .find_next('td') to get that next tag with the description:

import requests
import bs4

headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.76 Safari/537.36'} # This is chrome, you can set whatever browser you like
request = requests.get("https://www.cloudtango.org", verify=False, headers=headers)
soup = bs4.BeautifulSoup(request.content, 'html.parser')
data = soup.find_all('td', {'class':'company'})

for each in data:
    company =  each.find('img')['alt']   
    description = each.find_next('td').text
    print(f'{company}: {description}\n\n')

Output:

Redcentric: Redcentric is a leading UK IT managed services provider that offers a range of IT and Cloud services designed to support organisations in their journey from traditional infrastructure to the Cloud …


Modern Networks: Established in 1999, Modern Networks is a leading provider of IT support, network services, business broadband and telecoms to the UK’s commercial property sector. Additionally, we work with around …


BlackPoint IT Services: BlackPoint’s comprehensive range of Managed IT Services is designed to help you improve IT quality, efficiency and reliability -and save you up to 50% on IT cost. Providing IT solutions for more …


AffinityMSP: AffinityMSP was created with one goal in mind: to help Australian businesses achieve success through high-performance technology. Our consultants take the time to get to know your business and …


centrexIT: Founded in 2002, centrexIT is San Diego's leader in IT management. Our locally-based technology professionals provide outsourced IT service, support, security and leadership for small and medium-…


Carbon60: Carbon60 specializes in delivering secure managed cloud solutions for public and private sector organizations with business-critical workloads. Businesses are at different stages in their cloud …


...

Answered By - chitown88