The ingestion Pipeline
Training an AI chatbot on your website might seem like magic, but it involves a highly structured engineering pipeline behind the scenes. Here is how your website URL goes from a raw web link to semantic vectors.
1. Scraping and DOM Parsing
First, a server-side web scraper visits the URL. It downloads the raw HTML and parses the DOM. It ignores navigation bars, footers, and scripts, extracting only the main article content and descriptive headers.
2. Semantic Text Chunking
LLM prompt windows are limited, and long texts can dilute context. We slice the parsed HTML text into smaller pieces (e.g., 500-1000 characters). We use "smart chunking" to ensure we don't cut text mid-sentence, preserving complete semantic concepts.
3. Embedding Generation
Each text chunk is sent to an embedding model (like OpenAI text-embedding-3-small). The model returns a vector array representing the chunk's conceptual meaning. This vector is saved to a database table alongside the source URL reference.
When a user asks a question, we convert their query into a vector, find the closest matching chunks from your site, and feed them to the LLM to write a comprehensive answer.