Scraping Website: How to Write a Script in Python

Here's a python model script to scrape a website using the BeautifulSoup.

Python script

The logic below uses BeautifulSoup Package for web scraping.

import requests

from bs4 import BeautifulSoup

url = "https://www.example.com"

response = requests.get(url)

soup = BeautifulSoup(response.text, "html.parser")

# Print the title of the webpage

print(soup.title.text)

# Print all the links in the webpage

for link in soup.find_all("a"):

    print(link.get("href"))

In this script, we first import the Requests and Beautiful Soup libraries. We then define the URL we want to scrape and use the Requests library to send a GET request to that URL. We then pass the response text to Beautiful Soup to parse the HTML contents of the webpage.

We then use Beautiful Soup to extract the title of the webpage and print it to the console. We also use a for loop to find all the links in the webpage and print their href attributes to the console.

This is just a basic example, but you can use Beautiful Soup to extract any other HTML elements you want from the webpage.

References

• Python Automation Cookbook: 75 Python automation ideas for web scraping, data wrangling, and processing Excel, reports, emails, and more, 2nd Edition