Web Scraping for RAG: Clean Data Extraction at Scale

Raw HTML is full of nav bars, footer links, cookie banners, scripts, and layout wrappers — feed that to an LLM and you pay tokens for noise while retrieval quality collapses. For RAG, the output format and cleanliness matter more than the fetching: clean Markdown is dramatically more token-efficient than raw HTML and directly embeddable. The reference consensus is that cleaning (boilerplate removal) can improve retrieval accuracy by up to ~20%.

This post shows how to scrape JS-heavy sites at scale with Playwright and produce LLM-ready Markdown, chunked and metadated for a vector index.

Prerequisites

• Python 3.10+ and pip install playwright then playwright install chromium
• pip install readability-lxml markdownify (or beautifulsoup4)

Step 1: Render before you extract

A plain requests fetch returns the skeleton HTML — modern React/Vue SPAs populate content only after JavaScript runs. Playwright controls a real headless browser, so the fully rendered DOM is what you read:

import asyncio
from playwright.async_api import async_playwright

async def fetch_rendered(url: str) -> str:
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        page = await browser.new_page()
        await page.goto(url, wait_until="networkidle")  # wait for JS to settle
        await page.wait_for_selector("main, article, .content", timeout=5000).catch(lambda _: None)
        html = await page.content()
        await browser.close()
        return html

networkidle and wait_for_selector replace guessy time.sleep(5) — Playwright waits for the actual condition.

Step 2: Strip boilerplate

Now take the rendered HTML and isolate the real article, dropping navigation, sidebars, ads, and comment sections. Mozilla Readability (the engine behind Firefox Reader View) is the standard:

from readability import Document

def extract_article(html: str) -> str:
    doc = Document(html)
    return doc.summary(html_partial=True)   # isolated main-content HTML

Step 3: Convert to Markdown

Markdown is the ideal shape for an LLM: headings, lists, tables, and code blocks survive as semantics, and the structural tags disappear. markdownify does the DOM→Markdown conversion:

from markdownify import markdownify as md

def html_to_markdown(html: str) -> str:
    article = extract_article(html)
    return md(article, heading_style="atx", bullets="*")

Collapse the whitespace and escape Markdown-special characters so formatting collisions don’t corrupt the output.

Step 4: Crawl politely, at scale

Concurrency plus discipline: a bounded queue, exponential backoff on 403s, respect robots.txt, and don’t hammer a host. Keep the browser count bounded so memory doesn’t balloon.

import asyncio, time

async def crawl(urls: list[str], concurrency: int = 5) -> list[str]:
    sem = asyncio.Semaphore(concurrency)

    async def one(url: str):
        async with sem:
            for attempt in range(3):
                try:
                    html = await fetch_rendered(url)
                    return html_to_markdown(html)
                except Exception as e:
                    if attempt == 2:
                        return f"FAILED {url}: {e}"
                    await asyncio.sleep(2 ** attempt)   # backoff
    return await asyncio.gather(*(one(u) for u in urls))

Keep to ~5 concurrent requests per host and use a modern user agent — older agents are a common trigger for WAF blocks.

Step 5: Chunk and enrich metadata

Chunk by structure, not by character count: split at headers, paragraphs, and code blocks (respecting Markdown). Then attach metadata — source URL, title, publish date, domain — because that becomes the filter surface in your vector store (“docs from the last 6 months”).

import re

def chunk_markdown(md_text: str, max_tokens: int = 500) -> list[str]:
    sections = re.split(r"\n(?=#)", md_text)      # split at headings
    chunks, buf = [], ""
    for sec in sections:
        if len(buf) + len(sec) > max_tokens * 4:  # ~4 chars/token
            chunks.append(buf); buf = sec
        else:
            buf += "\n" + sec
    if buf: chunks.append(buf)
    return chunks

Putting It All Together

A reusable scraper that yields ready-to-embed chunks:

from dataclasses import dataclass, asdict

@dataclass
class Chunk:
    text: str
    url: str
    title: str

async def scrape_for_rag(url: str) -> list[dict]:
    html = await fetch_rendered(url)
    article = extract_article(html)
    markdown = html_to_markdown(article)
    title = re.search(r"<title>(.*?)</title>", html, re.S).group(1).strip()
    return [asdict(Chunk(t, url, title)) for t in chunk_markdown(markdown)]

# embed + upsert into pgvector/Qdrant, then query in your RAG pipeline

When DIY is not the answer

The DIY route (Playwright + Readability + markdownify) gives total control but you operate the browsers, retries, and anti-bot handling. If you hit Cloudflare-level protection or need to scrape thousands of pages reliably, an extraction API (Firecrawl, Crawl4AI’s hosted mode, etc.) or a classifier-first scraper becomes the right boundary — it renders, cleans, and returns Markdown/JSON so the pipeline stays an API call instead of a fleet of browsers.

Conclusion & Next Steps

Web content for RAG needs rendering for JS sites, Readability for boilerplate, Markdown for token efficiency, and structural chunking with metadata for retrieval. Next: add title/date metadata filtering to your queries, use a small pre-scan to skip boilerplate pages before rendering, and evaluate retrieval recall before and after cleaning to measure what the cleanup actually bought you.

References / Sources

• Playwright Python docs. https://playwright.dev/python/docs/intro
• Mozilla Readability. https://github.com/mozilla/readability
• Turndown / markdownify (HTML→Markdown). https://github.com/mixmark-io/turndown
• Crawl4AI (LLM-ready crawler built on Playwright). https://github.com/unclecode/crawl4ai
• How to scrape webpages to Markdown for AI. https://geekflare.com/guides/how-to-scrape-webpages-to-markdown/