Web Scrapping using Python

Photo by Chris Ried on Unsplash
  1. Pandas
  2. Beautiful Soup
  3. Selenium
pip install panda
pip install BeautifulSoup4
pip install selenium
  1. Import the Libraries
import pandas as pd
from bs4 import BeautifulSoup
from selenium import webdriver
productName = []
productPrice = []
driver = webdriver.Chrome(’Path to the chrome webdriver’)
driver.get("paste the url here")
content = driver.page_source
The Red highlighted area describes the respective class and tag of the section. In this case, the tag we are looking is <a>
The Red highlighted area indicates the class and tag of the Product name
soup = BeautifulSoup(content,'html.parser')
for a in soup.findAll('a',href =True, attrs={'class':'_31qSD5'}):
name = a.find('div' , attrs{'class' :'_31qSD5'})
price= a.find('div',attrs{'class':'_1vC4OE _2rQ-NK'})
productName.append(name.text)
productPrice.append(price.text)
df = pd.DataFrame({'Product Name':productName,'Price':productPrice})
df.to_csv('Products.csv',index= False, encoding = 'utf-8')
Results

--

--

--

Student | Motorsports Enthusiast | Ferrari Tifosi | Dev

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Google Summer of Code’18 with AOSSIE

Multitenancy, Contextual Design of Ngrx

KRL Testing Tab

The 4 Pillars Of Object Oriented Programming

NetSuite/Copper CRM Integration

Scaling Hybrid/Multi Cloud Infrastructure Efficiently at Enterprises — Q4 2021 EAB Meeting

Why does BGL need great Business Analysts?

How to get started with Unit Testing? Part #1

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Dhanush

Dhanush

Student | Motorsports Enthusiast | Ferrari Tifosi | Dev

More from Medium

Complete Guide to Data Cleaning with Python

ENEM Data Analysis Using Python

Python Data Model

How to easily work with Docker and Jupyter as a Data Scientist?