Scraping home

There's a lot to it.

For non-interactive web pages, I recommend BeautifulSoup.

Even if normal BeautifulSoup doesn't work, there are lots of ways to pretend to be a browser. Give those a try before you go for browser automation tools.

If you're scraping a site that does require a browser automation tool, I recommend Playwright over Selenium. It's newer and weird for a Python dev, but the documentation is great.