Improve RAG quality with custom web crawling for Amazon Bedrock

Amazon Web Services has published a technical guide detailing how to enhance Retrieval Augmented Generation (RAG) applications by preprocessing web content before ingestion. The tutorial, released in November 2026, explains how developers can use the open-source tool Crawl4AI to clean and structure data before feeding it into Amazon Bedrock Knowledge Bases. This approach targets scenarios where default crawlers fail to isolate relevant information or lack necessary metadata context.

What happened

Building effective RAG systems relies heavily on the quality of ingested data. Amazon Bedrock Knowledge Bases offers a managed service that handles indexing, storage, and retrieval, including a built-in web crawler for public sites. While this managed connector is sufficient for many use cases, it operates within strict boundaries. It respects robots.txt files and captures changes via scheduled syncs, but it does not allow deep customization of how page content is parsed or filtered before it enters the vector store.

For organizations requiring finer control, AWS proposes a hybrid workflow. Instead of relying solely on the managed connector, developers can insert a preparation step using Crawl4AI. This Python-based crawler extracts specific content regions, removes navigation boilerplate, and attaches custom metadata attributes. The processed data is then stored as Markdown files and JSON sidecars in Amazon S3, which serves as the data source for the knowledge base. This method ensures that only high-signal content reaches the embedding models, potentially improving answer relevance.

The provided solution outlines a batch ingestion process rather than real-time web access. It uses AWS Fargate to run the crawling tasks on a schedule, ensuring that the infrastructure scales automatically without manual server management. The workflow is designed to be extensible, allowing teams to define their own parsing rules and metadata schemas while leveraging Amazon Bedrock for the heavy lifting of chunking and embedding.

How it works

The core mechanism involves three distinct stages: extraction, preparation, and ingestion. First, an Amazon EventBridge rule triggers an AWS Fargate task running a containerized Crawl4AI script. The crawler visits specified seed URLs and applies a breadth-first search strategy with configurable depth and page limits. It uses a pruning content filter to strip away headers, footers, and ads, retaining only the main article body, headings, and tables. The output is converted into clean Markdown, which preserves structural semantics better than raw HTML for large language models.

Next, the workflow generates a metadata sidecar for each document. This JSON file contains attributes such as the source URL, crawl timestamp, and custom categories. These sidecars are uploaded to Amazon S3 alongside the Markdown files, following a strict naming convention where the metadata file name matches the document name with a .metadata.json extension. Amazon Bedrock reads these sidecars during ingestion, enabling users to filter search results based on custom attributes like date or content type.

Finally, the task triggers an ingestion job in Amazon Bedrock Knowledge Bases. The service parses the Markdown, applies the selected chunking strategy, and creates vector embeddings. Because the content is already cleaned and structured, the chunking process is more likely to produce coherent segments. Users can choose between default, fixed-size, or no-chunking strategies depending on their specific retrieval needs. The system then indexes these vectors, making them available for query time retrieval with citations.

Key details

• Tooling: The solution uses Crawl4AI, an open-source Python crawler, integrated with Amazon Bedrock Knowledge Bases via Amazon S3.
• Data Format: Content is converted to Markdown to preserve structure while removing noise, accompanied by JSON metadata sidecars.
• Infrastructure: The crawling process runs on AWS Fargate, triggered by Amazon EventBridge on a configurable schedule.
• Metadata Limits: Each metadata sidecar must remain under 10 KB; exceeding this limit causes the upload to fail.
• Chunking Strategies: Users can select default, fixed-size, or no-chunking options during data source configuration, but cannot change them later without creating a new source.
• Authentication: The sample implementation supports only public websites; authenticated sites require additional security reviews and session handling.

Why it matters

For software engineers building AI products, the difference between a useful answer and a hallucination often lies in data hygiene. Default web crawlers tend to ingest entire pages, including navigation menus, cookie banners, and footer links. This noise dilutes the semantic meaning of the content, leading to poorer retrieval scores. By preprocessing data with tools like Crawl4AI, teams can ensure that the vector store contains only the information users actually care about. This leads to more accurate responses and reduces the token count required for context windows.

Additionally, the ability to attach custom metadata transforms a simple search engine into a filtered database. Developers can tag documents with expiration dates, department codes, or confidence scores. At query time, these tags allow for precise filtering, ensuring that the AI only references current or authorized information. This level of control is critical for enterprise applications where compliance and accuracy are non-negotiable. The modular nature of this approach also means teams can swap out the crawler or the embedding model without rewriting the entire application logic.

What you can do

• Evaluate your current RAG pipeline to identify if noisy web content is degrading answer quality.
• Clone the aws-samples repository for Crawl4AI and Amazon Bedrock to test the workflow in a sandbox environment.
• Define clear CSS selectors or pruning rules to isolate the main content areas of your target websites.
• Design a metadata schema that includes useful filters like publication date, author, or document category.
• Test different chunking strategies in Amazon Bedrock using representative samples to find the optimal balance between context and precision.
• Implement strict URL scoping and rate limiting in your crawler configuration to respect website terms of service.