[FIXED] Unable to scrape every table from webpage

Issue

I need to create a csv with every RBAC action listed on this Azure webpage - https://learn.microsoft.com/en-us/azure/role-based-access-control/resource-provider-operations

I have two problems.

1. I can't get all the tables, only a few towards the end
2. The small amount I can scrape isn't being loaded as csv format

Here's my code:

import requests
from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
import pandas as pd
import csv

# headless background execution
Options = Options()
Options.headless = True

url = "https://learn.microsoft.com/en-us/azure/role-based-access-control/resource-provider-operations"
browser = webdriver.Chrome(options=Options)
browser.get(url)

soup = BeautifulSoup(requests.get(url).content, 'lxml')
tables = soup.findAll("table")

with open('output.csv', 'w', newline='') as csvfile:

    propertyWriter = csv.writer(csvfile, quoting=csv.QUOTE_ALL)

    for listing in tables:
        propertyWriter.writerow(listing)

Solution

With your code, I think you can get all tables:

dfs = []
for table in tables[1:]:  # skip the toc
    df = pd.read_html(io.StringIO(str(table)))[0]
    dfs.append(df)
df = pd.concat(dfs, ignore_index=True)
df.to_csv('output.csv', index=False)

Output:

>>> df
                                                  Action                                        Description
0                       Microsoft.Addons/register/action  Register the specified subscription with Micro...
1                       Microsoft.Addons/operations/read                      Gets supported RP operations.
2      Microsoft.Addons/supportProviders/listsupportp...  Lists current support plan information for the...
3      Microsoft.Addons/supportProviders/supportPlanT...    Get the specified Canonical support plan state.
4      Microsoft.Addons/supportProviders/supportPlanT...    Adds the Canonical support plan type specified.
...                                                  ...                                                ...
13587              Microsoft.ServicesHub/connectors/read               View or List Services Hub Connectors
13588            Microsoft.ServicesHub/connectors/delete                     Delete Services Hub Connectors
13589  Microsoft.ServicesHub/connectors/checkAssessme...  Lists the Assessment Entitlements for a given ...
13590  Microsoft.ServicesHub/supportOfferingEntitleme...  View the Support Offering Entitlements for a g...
13591              Microsoft.ServicesHub/workspaces/read  List the Services Hub Workspaces for a given User

[13592 rows x 2 columns]

Answered By - Corralien