Issue
My code is supposed to pull data from a website using the beautifulsoup library, but keeps on returning none as the output when I try to use it.
my code
import pandas as pd
import datetime
import requests
from requests.exceptions import ConnectionError
from bs4 import BeautifulSoup
url = 'https://finance.yahoo.com/quote/' + stock_code + '?p=' + stock_code
r = requests.get(url)
web_content = BeautifulSoup(r.text, 'lxml')
#texts = web_content_div(web_content, 'My(6px) Pos(r) smartphone_Mt(6px) W(100%)')
section = web_content.find('fin-streamer')
display(section)
is trying to pull a number from yahoo finance shown here
it keeps returning none, frankly this is probably just a formatting issue, but it's 1 and I want to go to sleep. Any advice on how to actually pull what I want from the website?
Solution
I think using your original requests.get
call won't return the same result you get from the browser due to being blocked by Yahoo Finance. To workaround it, you should add the parameter headers
with user-agent specified (headers={'USER-AGENT': "Mozilla/5.0"}
). Afterwards, if you print out the response text content, you should see there are multiple fin-streamer
tags in the html text
import pandas as pd
import datetime
import requests
from requests.exceptions import ConnectionError
from bs4 import BeautifulSoup
stock_code = 'AAPL'
url = 'https://finance.yahoo.com/quote/' + stock_code + '?p=' + stock_code
r = requests.get(url, headers={'USER-AGENT': "Mozilla/5.0"})
web_content = BeautifulSoup(r.text, 'lxml')t.find('fin-streamer')
print(section)
Answered By - tax evader
0 comments:
Post a Comment
Note: Only a member of this blog may post a comment.