[FIXED] BeautifulSoup not getting all HTML elements anymore while it did yesterday and no code was changed

Issue

The source variable is not getting all the HTMl elements but it was before. Nothing was changed and I don't know what to do. I'm very new to coding. I'm trying to extract a list of songs and artists off my apple music playlist and before this worked fine but now the HTML is not appearing as it does on inspect element on the website.

from bs4 import BeautifulSoup
import requests

source = requests.get('https://music.apple.com/us/playlist/vibe-check-1/pl.u-JPAZE8Gul2rRBZ0').text

soup = BeautifulSoup(source, 'lxml')

print(soup.prettify()) 

links = soup.find('div', {"class":"songs-list typography-caption"})


song_names = links.find_all('div', {"class": "song-name typography-label"})
artist_name = links.find_all('div', {"class":"by-line typography-caption"})

Solution

The data is loaded dynamically, therefore requests doesn't support it. However, it's possible to scrape the data by sending a GET request to their API with adding the correct authorization.

There's also no need for BeautifulSoup, this can be done with just the requests module.

import requests


URL = "https://amp-api.music.apple.com/v1/catalog/us/playlists/pl.u-JPAZE8Gul2rRBZ0?omit%5Bresource%5D=autos&views=featured-artists&extend=artistUrl%2CcomposerUrl%2CtrackCount%2CeditorialVideo&include=tracks&include%5Bplaylists%5D=curator&include%5Bsongs%5D=artists%2Ccomposers&fields%5Bartists%5D=name%2Curl%2Cartwork&art%5Burl%5D=f&l=en-us"

HEADERS = {
    "authorization": "Bearer eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCIsImtpZCI6IldlYlBsYXlLaWQifQ.eyJpc3MiOiJBTVBXZWJQbGF5IiwiaWF0IjoxNjA3OTcyMDQ5LCJleHAiOjE2MjM1MjQwNDl9.hHSg1_2xr_-Gj8ZWIhoRARUEyTygMYUlvgfunJr4PsGikIylISAsaqx25gBuOBVKnSTl2btA22FsYM0OUBNU3A",
    "Referer": "https://music.apple.com/",
}

response = requests.get(URL, headers=HEADERS).json()

for data in response["data"][0]["relationships"]["tracks"]["data"]:
    print("Song name: ", data["attributes"]["name"])
    print("Artist:", data["attributes"]["artistName"] + "\n")

Output:

Song name:  Sky Walker (feat. Travis Scott)
Artist: Miguel

Song name:  Location
Artist: Khalid

...And on...

Answered By - MendelG