[FIXED] Problem scraping webpage using beautifulsoup and requests

Issue

I was previously able to use beautifulsoup and requests to scrape and parse this page:

https://www.fangraphs.com/leaders/major-league?pos=all&stats=pit&lg=all&qual=y&type=36&season=2023&month=0&season1=2023&ind=0

with this code:

import pandas as pd
import requests
from datetime import date, timedelta
from bs4 import BeautifulSoup
import lxml
import numpy as np

def parse_array_from_fangraphs_html(start_date,end_date, URL_1):
    """
    Take a HTML stats page from fangraphs and parse it out to a 
    dataframe.
    """
    # parse input
    PITCHERS_URL = URL_1
    # request the data
    pitchers_html = requests.get(PITCHERS_URL).text
    soup = BeautifulSoup(pitchers_html, "lxml")
    table = soup.find("table", {"class": "rgMasterTable"})

    # get headers
    headers_html = table.find("thead").find_all("th")
    headers = []
    for header in headers_html:
        headers.append(header.text)

    # get rows
    rows = []
    rows_html = table.find("tbody").find_all("tr")
    for row in rows_html:
        row_data = []
        for cell in row.find_all("td"):
            row_data.append(cell.text)
        rows.append(row_data)

    return pd.DataFrame(rows, columns = headers)

sdate = '2022-01-01'
enddate = date.today()
enddate =enddate.strftime("%Y-%m-%d")
PITCHERS = "https://www.fangraphs.com/leaders/major-league?pos=all&stats=pit&lg=all&qual=y&type=36&season=2023&month=0&season1=2023&ind=0"

wRC1 = parse_array_from_fangraphs_html(sdate, enddate, PITCHERS)

But something changed and it is not working anymore. Can I get an idea of what happened and how can I do it now? I want to scrape data from the table, and put it into a dataframe.

Thanks in advance, Carlos.

Solution

If you want that data from the table, you may want to:

• inspect XHR calls in browser's Dev tools -- Network tab
• scrape the API endpoint directly

Like below:

import requests
import pandas as pd
headers= {
    'User-Agent':'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36'
}

api_url = 'https://www.fangraphs.com/api/leaders/major-league/data?age=&pos=all&stats=pit&lg=all&qual=y&season=2023&season1=2023&startdate=2023-03-01&enddate=2023-11-01&month=0&hand=&team=0&pageitems=99&pagenum=1&ind=0&rost=0&players=&type=36&sortdir=default&sortstat=sp_pitching'
r = requests.get(api_url, headers=headers)
df = pd.json_normalize(r.json(), record_path=['data'])
print(df)

Result in terminal:

    Throws  xMLBAMID    Name    Team    Season  Age     AgeR    SeasonMin   SeasonMax   W   ...     HardHit%    Q   TG  TIP     PlayerNameRoute     PlayerName  playerid    TeamName    TeamNameAbb     teamid
0   R   675911  <a href="statss.aspx?playerid=27498&position=P...   <a href="leaders.aspx?pos=all&stats=pit&lg=all...   2023    24.0    24 - 24     2023    2023    18.0    ...     0.347368    1.16556     151.000000  176.000000  Spencer Strider     Spencer Strider     27498   ATL     ATL     16
1   R   554430  <a href="statss.aspx?playerid=10310&position=P...   <a href="leaders.aspx?pos=all&stats=pit&lg=all...   2023    33.0    33 - 33     2023    2023    12.0    ...     0.366397    1.19868     151.000000  181.000000  Zack Wheeler    Zack Wheeler    10310   PHI     PHI     26
2   R   669923  <a href="statss.aspx?playerid=25436&position=P...   <a href="leaders.aspx?pos=all&stats=pit&lg=all...   2023    25.0    25 - 25     2023    2023    10.0    ...     0.412698    1.13687     151.000000  171.666656  George Kirby    George Kirby    25436   SEA     SEA     11
3   R   641154  <a href="statss.aspx?playerid=17085&position=P...   <a href="leaders.aspx?pos=all&stats=pit&lg=all...   2023    27.0    27 - 27     2023    2023    10.0    ...     0.348548    1.20833     152.000000  183.666656  Pablo Lopez     Pablo López     17085   MIN     MIN     8
4   R   543037  <a href="statss.aspx?playerid=13125&position=P...   <a href="leaders.aspx?pos=all&stats=pit&lg=all...   2023    32.0    32 - 32     2023    2023    13.0    ...     0.395626    1.27152     151.000000  192.000000  Gerrit Cole     Gerrit Cole     13125   NYY     NYY     9
5   R   645261  <a href="statss.aspx?playerid=18684&position=P...   <a href="leaders.aspx?pos=all&stats=pit&lg=all...   2023    27.0    27 - 27     2023    2023    7.0     ...     0.401434    1.21491     152.000000  184.666656  Sandy Alcantara     Sandy Alcantara     18684   MIA     MIA     20
6   R   668678  <a href="statss.aspx?playerid=19291&position=P...   <a href="leaders.aspx?pos=all&stats=pit&lg=all...   2023    27.0    27 - 27     2023    2023    16.0    ...     0.456722    1.30044     152.000000  197.666656  Zac Gallen  Zac Gallen  19291   ARI     ARI     15
7   R   669203  <a href="statss.aspx?playerid=19361&position=P...   <a href="leaders.aspx?pos=all&stats=pit&lg=all...   2023    28.0    28 - 28     2023    2023    9.0     ...     0.326489    1.22296     151.000000  184.666656  Corbin Burnes   Corbin Burnes   19361   MIL     MIL     23
8   R   621107  <a href="statss.aspx?playerid=13774&position=P...   <a href="leaders.aspx?pos=all&stats=pit&lg=all...   2023    29.0    29 - 29     2023    2023    15.0    ...     0.347732    1.10307     152.000000  167.666672  Zach Eflin  Zach Eflin  13774   TBR     TBR     12
9   R   605400  <a href="statss.aspx?playerid=16149&position=P...   <a href="leaders.aspx?pos=all&stats=pit&lg=all...   2023    30.0    30 - 30     2023    2023    12.0    ...     0.381323    1.19868     151.000000  181.000000  Aaron Nola  Aaron Nola  16149   PHI     PHI     26
10  R   592332  <a href="statss.aspx?playerid=14107&position=P...   <a href="leaders.aspx?pos=all&stats=pit&lg=all...   2023    32.0    32 - 32     2023    2023    11.0    ...     0.432802    1.13907     151.000000  172.000000  Kevin Gausman   Kevin Gausman   14107   TOR     TOR     14
11  R   642547  <a href="statss.aspx?playerid=18679&position=P...   <a href="leaders.aspx?pos=all&stats=pit&lg=all...   2023    27.0    27 - 27     2023    2023    12.0    ...     0.370277    1.07726     151.000000  162.666656  Freddy Peralta  Freddy Peralta  18679   MIL     MIL     23
12  R   680694  <a href="statss.aspx?playerid=24586&position=P...   <a href="leaders.aspx?pos=all&stats=pit&lg=all...   2023    26.0    26 - 26     2023    2023    11.0    ...     0.433014    1.01104     151.000000  152.666656  Kyle Bradish    Kyle Bradish    24586   BAL     BAL     2
13  R   657376  <a href="statss.aspx?playerid=19899&position=P...   <a href="leaders.aspx?pos=all&stats=pit&lg=all...   2023    27.0    27 - 27     2023    2023    9.0     ...     0.413043    1.00000     151.000000  150.999994  Clarke Schmidt  Clarke Schmidt  19899   NYY     NYY     9
14  R   656605  <a href="statss.aspx?playerid=17594&position=P...   <a href="leaders.aspx?pos=all&stats=pit&lg=all...   2023    27.0    27 - 27     2023    2023    12.0    ...     0.339806    1.20971     151.000000  182.666672  Mitch Keller    Mitch Keller    17594   PIT     PIT     27
15  R   543243  <a href="statss.aspx?playerid=12768&position=P...   <a href="leaders.aspx?pos=all&stats=pit&lg=all...   2023    33.0    33 - 33     2023    2023    8.0     ...     0.394628    1.14474     152.000000  174.000000  Sonny Gray  Sonny Gray  12768   MIN     MIN     8
16  R   453286  <a href="statss.aspx?playerid=3137&position=P"...   - - -   2023    38.0    38 - 38     2023    2023    13.0    ...     0.369347    1.01104     151.000000  152.666664  Max Scherzer    Max Scherzer    3137    - - -   2 Tms   0
17  R   656302  <a href="statss.aspx?playerid=18525&position=P...   <a href="leaders.aspx?pos=all&stats=pit&lg=all...   2023    27.0    27 - 27     2023    2023    7.0     ...     0.421634    1.08553     152.000000  165.000000  Dylan Cease     Dylan Cease     18525   CHW     CHW     4
18  R   518876  <a href="statss.aspx?playerid=11156&position=P...   <a href="leaders.aspx?pos=all&stats=pit&lg=all...   2023    34.0    34 - 34     2023    2023    11.0    ...     0.402913    1.04605     152.000000  159.000000  Merrill Kelly   Merrill Kelly   11156   ARI     ARI     15
19  R   621244  <a href="statss.aspx?playerid=14168&position=P...   <a href="leaders.aspx?pos=all&stats=pit&lg=all...   2023    29.0    29 - 29     2023    2023    11.0    ...     0.353057    1.17881     151.000000  178.000000  Jose Berrios    José Berríos    14168   TOR     TOR     14
20  L   579328  <a href="statss.aspx?playerid=20633&position=P...   <a href="leaders.aspx?pos=all&stats=pit&lg=all...   2023    32.0    32 - 32     2023    2023    10.0    ...     0.417234    1.05077     151.000000  158.666656  Yusei Kikuchi   Yusei Kikuchi   20633   TOR     TOR     14
21  R   657277  <a href="statss.aspx?playerid=17995&position=P...   <a href="leaders.aspx?pos=all&stats=pit&lg=all...   2023    26.0    26 - 26     2023    2023    10.0    ...     0.452007    1.33113     151.000000  201.000000  Logan Webb  Logan Webb  17995   SFG     SFG     30
22  R   669302  <a href="statss.aspx?playerid=22250&position=P...   <a href="leaders.aspx?pos=all&stats=pit&lg=all...   2023    26.0    26 - 26     2023    2023    13.0    ...     0.449704    1.18543     151.000000  179.000000  Logan Gilbert   Logan Gilbert   22250   SEA     SEA     11
23  L   657006  <a href="statss.aspx?playerid=17312&position=P...   <a href="leaders.aspx?pos=all&stats=pit&lg=all...   2023    27.0    27 - 27     2023    2023    16.0    ...     0.363057    1.09272     151.000000  165.000000  Justin Steele   Justin Steele   17312   CHC     CHC     17
24  R   622491  <a href="statss.aspx?playerid=15689&position=P...   <a href="leaders.aspx?pos=all&stats=pit&lg=all...   2023    30.0    30 - 30     2023    2023    14.0    ...     0.421371    1.24724     151.000000  188.333328  Luis Castillo   Luis Castillo   15689   SEA     SEA     11
25  L   666200  <a href="statss.aspx?playerid=19959&position=P...   <a href="leaders.aspx?pos=all&stats=pit&lg=all...   2023    25.0    25 - 25     2023    2023    10.0    ...     0.401806    1.09430     152.000000  166.333328  Jesus Luzardo   Jesús Luzardo   19959   MIA     MIA     20
26  R   608337  <a href="statss.aspx?playerid=15474&position=P...   - - -   2023    28.0    28 - 28     2023    2023    8.0     ...     0.415948    1.12527     151.666667  170.666664  Lucas Giolito   Lucas Giolito   15474   - - -   3 Tms   0
27  R   458681  <a href="statss.aspx?playerid=2520&position=P"...   - - -   2023    36.0    36 - 36     2023    2023    12.0    ...     0.390196    1.13687     151.000000  171.666664  Lance Lynn  Lance Lynn  2520    - - -   2 Tms   0
28  R   571945  <a href="statss.aspx?playerid=9803&position=P"...   <a href="leaders.aspx?pos=all&stats=pit&lg=all...   2023    34.0    34 - 34     2023    2023    7.0     ...     0.425566    1.20751     151.000000  182.333328  Miles Mikolas   Miles Mikolas   9803    STL     STL     28
29  R   502171  <a href="statss.aspx?playerid=6562&position=P"...   <a href="leaders.aspx?pos=all&stats=pit&lg=all...   2023    35.0    35 - 35     2023    2023    7.0     ...     0.432203    1.00221     151.000000  151.333328  Alex Cobb   Alex Cobb   6562    SFG     SFG     30
30  R   665152  <a href="statss.aspx?playerid=19350&position=P...   <a href="leaders.aspx?pos=all&stats=pit&lg=all...   2023    27.0    27 - 27     2023    2023    12.0    ...     0.413442    1.08609     151.000000  164.000000  Dean Kremer     Dean Kremer     19350   BAL     BAL     2
31  L   605483  <a href="statss.aspx?playerid=13543&position=P...   <a href="leaders.aspx?pos=all&stats=pit&lg=all...   2023    30.0    30 - 30     2023    2023    14.0    ...     0.342711    1.14474     152.000000  174.000000  Blake Snell     Blake Snell     13543   SDP     SDP     29
32  L   676664  <a href="statss.aspx?playerid=23429&position=P...   <a href="leaders.aspx?pos=all&stats=pit&lg=all...   2023    27.0    27 - 27     2023    2023    5.0     ...     0.373695    1.08168     151.000000  163.333328  JP Sears    JP Sears    23429   OAK     OAK     10
33  L   664285  <a href="statss.aspx?playerid=17295&position=P...   <a href="leaders.aspx?pos=all&stats=pit&lg=all...   2023    29.0    29 - 29     2023    2023    12.0    ...     0.448940    1.24123     152.000000  188.666672  Framber Valdez  Framber Valdez  17295   HOU     HOU     21
34  R   450203  <a href="statss.aspx?playerid=4676&position=P"...   <a href="leaders.aspx?pos=all&stats=pit&lg=all...   2023    39.0    39 - 39     2023    2023    14.0    ...     0.387615    1.07506     151.000000  162.333328  Charlie Morton  Charlie Morton  4676    ATL     ATL     16
35  R   605135  <a href="statss.aspx?playerid=12304&position=P...   <a href="leaders.aspx?pos=all&stats=pit&lg=all...   2023    34.0    34 - 34     2023    2023    14.0    ...     0.351402    1.22958     151.000000  185.666656  Chris Bassitt   Chris Bassitt   12304   TOR     TOR     14
36  R   543475  <a href="statss.aspx?playerid=7593&position=P"...   <a href="leaders.aspx?pos=all&stats=pit&lg=all...   2023    32.0    32 - 32     2023    2023    4.0     ...     0.350459    1.09649     152.000000  166.666656  Jordan Lyles    Jordan Lyles    7593    KCR     KCR     7
37  R   592836  <a href="statss.aspx?playerid=11836&position=P...   <a href="leaders.aspx?pos=all&stats=pit&lg=all...   2023    30.0    30 - 30     2023    2023    15.0    ...     0.405462    1.05740     151.000000  159.666656  Taijuan Walker  Taijuan Walker  11836   PHI     PHI     26
38  L   666129  <a href="statss.aspx?playerid=21844&position=P...   <a href="leaders.aspx?pos=all&stats=pit&lg=all...   2023    25.0    25 - 25     2023    2023    9.0     ...     0.448889    1.02412     152.000000  155.666656  Braxton Garrett     Braxton Garrett     21844   MIA     MIA     20
39  R   502043  <a href="statss.aspx?playerid=10123&position=P...   <a href="leaders.aspx?pos=all&stats=pit&lg=all...   2023    35.0    35 - 35     2023    2023    14.0    ...     0.446655    1.19205     151.000000  180.000000  Kyle Gibson     Kyle Gibson     10123   BAL     BAL     2
40  R   673540  <a href="statss.aspx?playerid=31838&position=P...   <a href="leaders.aspx?pos=all&stats=pit&lg=all...   2023    30.0    30 - 30     2023    2023    11.0    ...     0.394737    1.02870     151.000000  155.333328  Kodai Senga     Kodai Senga     31838   NYM     NYM     25
41  L   656756  <a href="statss.aspx?playerid=16511&position=P...   - - -   2023    30.0    30 - 30     2023    2023    9.0     ...     0.373563    1.16336     151.000000  175.666664  Jordan Montgomery   Jordan Montgomery   16511   - - -   2 Tms   0
42  R   693821  <a href="statss.aspx?playerid=27779&position=P...   <a href="leaders.aspx?pos=all&stats=pit&lg=all...   2023    24.0    24 - 24     2023    2023    12.0    ...     0.404762    1.10817     151.000000  167.333328  Bryce Elder     Bryce Elder     27779   ATL     ATL     16
43  R   670912  <a href="statss.aspx?playerid=22487&position=P...   <a href="leaders.aspx?pos=all&stats=pit&lg=all...   2023    25.0    25 - 25     2023    2023    8.0     ...     0.391919    1.10375     151.000000  166.666656  Johan Oviedo    Johan Oviedo    22487   PIT     PIT     27
44  R   641540  <a href="statss.aspx?playerid=19409&position=P...   <a href="leaders.aspx?pos=all&stats=pit&lg=all...   2023    28.0    28 - 28     2023    2023    10.0    ...     0.411017    1.03974     151.000000  156.999992  Dane Dunning    Dane Dunning    19409   TEX     TEX     13
45  R   663903  <a href="statss.aspx?playerid=25377&position=P...   <a href="leaders.aspx?pos=all&stats=pit&lg=all...   2023    26.0    26 - 26     2023    2023    8.0     ...     0.484375    1.05044     152.000000  159.666656  Brady Singer    Brady Singer    25377   KCR     KCR     7
46  L   571578  <a href="statss.aspx?playerid=9323&position=P"...   <a href="leaders.aspx?pos=all&stats=pit&lg=all...   2023    33.0    33 - 33     2023    2023    10.0    ...     0.434326    1.12500     152.000000  171.000000  Patrick Corbin  Patrick Corbin  9323    WSN     WSN     24
47  L   607536  <a href="statss.aspx?playerid=16256&position=P...   <a href="leaders.aspx?pos=all&stats=pit&lg=all...   2023    30.0    30 - 30     2023    2023    6.0     ...     0.433333    1.03091     151.000000  155.666656  Kyle Freeland   Kyle Freeland   16256   COL     COL     19

48 rows × 430 columns

​ Please check Pandas documentation, as well as Requests documentation.

Answered By - Barry the Platipus