Advertisement

Python Project with source Code Web Scraper with Flask Frontend

Yuvraj singh July 23, 2024
Python Project with source Code Web Scraper with Flask Frontend

Building a web scraper with a Flask frontend can be a great intermediate Python project. It involves fetching data from websites using scraping libraries like BeautifulSoup or Scrapy, and then presenting this data in a web interface created using Flask.

Project Outline

  1. Web Scraping Component

    • Choose a website to scrape (ensure you have permission and comply with their terms of service).
    • Use libraries like BeautifulSoup or Scrapy to extract data from HTML pages.
    • Structure your scraper to fetch specific data points (e.g., headlines, prices, descriptions).

  2. Flask Frontend

    • Create a Flask application that will serve as the frontend.
    • Design HTML templates to display the scraped data in a user-friendly format.
    • Use Flask routes to handle different URLs and render appropriate templates.

  3. Integration

    • Integrate the scraping logic with your Flask application.
    • Ensure error handling for cases like failed requests or parsing errors.
    • Display the scraped data on your Flask web interface.

Example Source Code

Here's an example project structure and source code for a simple web scraper with a Flask frontend

1. Project Structure

├── app/
│   ├── static/
│   │   └── style.css
│   ├── templates/
│   │   ├── index.html
│   │   └── results.html
│   ├── __init__.py
│   └── scraper.py
├── requirements.txt
└── run.py

2. Flask Application (app/__init__.py)

from flask import Flask, render_template, request
from .scraper import scrape_data

app = Flask(__name__)

@app.route('/')
def index():
    return render_template('index.html')

@app.route('/results', methods=['POST'])
def results():
    url = request.form['url']
    data = scrape_data(url)
    return render_template('results.html', data=data)

if __name__ == '__main__':
    app.run(debug=True)

3. Web Scraper (app/scraper.py)

import requests
from bs4 import BeautifulSoup

def scrape_data(url):
    # Example: Scraping headlines from a news website
    headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36'}
    response = requests.get(url, headers=headers)
    
    if response.status_code == 200:
        soup = BeautifulSoup(response.content, 'html.parser')
        headlines = [headline.text.strip() for headline in soup.find_all('h2')]
        return headlines
    else:
        return None

4. HTML Templates (app/templates/index.html and app/templates/results.html)

index.html

<!DOCTYPE html>

<html lang="en">

<head>

    <meta charset="UTF-8">

    <meta name="viewport" content="width=device-width, initial-scale=1.0">

    <link rel="stylesheet" href="{{ url_for('static', filename='style.css') }}">

    <title>Web Scraper</title>

</head>

<body>

    <h1>Web Scraper</h1>

    <form action="/results" method="post">

        <label for="url">Enter URL to Scrape:</label>

        <input type="text" id="url" name="url" required>

        <button type="submit">Scrape</button>

    </form>

</body>

</html>

results.html

<!DOCTYPE html>

<html lang="en">

<head>

    <meta charset="UTF-8">

    <meta name="viewport" content="width=device-width, initial-scale=1.0">

    <link rel="stylesheet" href="{{ url_for('static', filename='style.css') }}">

    <title>Scraped Results</title>

</head>

<body>

    <h1>Scraped Results</h1>

    <ul>

        {% for headline in data %}

            <li>{{ headline }}</li>

        {% endfor %}

    </ul>

</body>

</html>

Styling (app/static/style.css)

body {

    font-family: Arial, sans-serif;

    margin: 20px;

}


h1 {

    color: #333;

}


form {

    margin-bottom: 20px;

}


ul {

    list-style-type: none;

    padding: 0;

}


li {

    margin-bottom: 10px;

}


Running the Application (run.py)

from app import app

if __name__ == '__main__':
    app.run(debug=True)

How to Run

  1. Clone the repository or create the directory structure as shown above.
  2. Install dependencies from requirements.txt (Flask, requests, beautifulsoup4).
  3. Run the Flask application by executing python run.py in your terminal.
  4. Access the application in your web browser at http://localhost:5000.

Notes

  • Security: Ensure that the websites you scrape permit web scraping and that you respect their robots.txt file.
  • Error Handling: Implement robust error handling to manage cases like network errors or incorrect URLs.
  • Deployment: For production, consider deploying on platforms like Heroku or PythonAnywhere.

This project combines web scraping, frontend development with Flask, and basic web application architecture, making it an excellent intermediate-level Python project.


Tags
Reactions

Post a Comment

0 Comments