What actually happens between pasting a URL and getting an answer
Five stages: discover, extract, chunk, embed, index. Then, on every question, a retrieval step that puts your own passages in front of the model before it writes anything. Here it is in full, because knowing the mechanism is what lets you fix a bad answer.
- Stage 1Discover
Find the pages worth reading
You give it a URL. CustomerBot follows the links it finds and returns the list of reachable pages, which you review before anything is trained. This matters more than it sounds: a blog archive with 400 posts will happily bury the six pages that answer real questions.
You pick the URLs. Nothing is trained without you seeing the list.
- Stage 2Extract
Pull the text out of the markup
Each selected page is fetched and reduced to its readable content — navigation, footers and boilerplate are not what you want retrieved when someone asks about your refund window.
PDFs go through the same path, including text recovered from image-heavy pages.
- Stage 3Chunk
Split it into passages
A whole page is the wrong unit. Retrieval works on passages, so pages are split into overlapping chunks small enough to be specific and large enough to keep a thought intact. Overlap is what stops an answer being cut in half at a chunk boundary.
This is why well-structured pages with real headings retrieve better than walls of text.
- Stage 4Embed
Turn each passage into a vector
Every chunk is run through an embedding model and becomes a high-dimensional vector — a numerical position that encodes meaning rather than words. Two passages that say the same thing in different vocabulary end up near each other.
Meaning, not keywords. “Why am I being charged twice” finds your billing-cycle page.
- Stage 5Index
Store them for fast similarity search
Vectors are written to a vector database, where finding the closest passages to a question is a cosine-similarity search rather than a scan. That is what keeps retrieval fast enough to sit inside a live chat.
Re-training a URL replaces that page’s vectors rather than adding a second copy.
And then, on every single question
Training happens once. Retrieval happens every time, which is why updating a page is enough to change what the assistant says.
- 01
Visitor asks
“Do you ship to Ireland and how long does it take?”
- 02
Question is embedded
The same model turns the question into a vector.
- 03
Nearest passages retrieved
The shipping-zones and delivery-times chunks come back highest.
- 04
Model reads, then writes
Those passages go into the prompt. The model answers from them, not from memory.
- 05
Source attached
The reply cites the page it used, so a wrong answer is traceable to a wrong page.
Train on less than you think
The instinct is to feed it everything. In practice, precision beats volume: irrelevant passages compete with the right ones for the same retrieval slots.
Worth training on
- Pricing, plans and what each tier includes
- Shipping, delivery and returns policies
- Setup, onboarding and how-to guides
- FAQ pages you already maintain
- Terms that customers actually ask about
- Contact details, hours and locations
Usually noise
- News posts and press releases
- Blog archives older than your current product
- Author bios and tag pages
- Anything superseded but not deleted
- Login-walled pages the crawler cannot reach anyway
- Duplicate content across regional subfolders
Every answer carries its source
An answer without a source is a rumour. Because retrieval knows exactly which passages produced a reply, the assistant can show them — which makes debugging a bad answer a matter of looking at what was retrieved rather than guessing at a black box.
Fixing an answer means fixing a page
There is no fine-tune to redo and no model to retrain. Edit the page, re-train that URL, and the next question is answered from the new text. If the page was always ambiguous, that is the real bug — and now you know.
Questions about training
How long does crawling take?
Minutes for a normal marketing site. Discovery is fast; the slow part is embedding, which scales with how much text you selected. Most people are asking their assistant real questions within a couple of minutes of pasting a URL.
Does it re-crawl automatically when I update a page?
No — training is a deliberate action. Re-train the URL from the training screen and that page’s passages are replaced. Making it part of your publishing checklist takes ten seconds and avoids the surprise of a bot quoting a page you rewrote last month.
How much content can I train on?
Storage is 5 MB per assistant on the free plan and 40 MB on paid plans. Because pages are stored as extracted text rather than as page images, that is far more content than it sounds — a large documentation site fits comfortably.
What if my site is JavaScript-rendered?
Content that only exists after client-side rendering may not be visible to the crawler. If a page looks empty after training, the reliable workaround is to paste the text directly as a text source, or upload the equivalent PDF.
Can it crawl pages behind a login?
No. The crawler sees your site the way an anonymous visitor does. For gated content, paste the text or upload the document instead.
What happens when retrieval finds nothing relevant?
The assistant falls back rather than inventing an answer, and the question is recorded. That list of unanswered questions is the most useful output of the whole system — it is your visitors telling you which page to write next.
Paste a URL and watch it happen
The crawl, the page picker and the first real answer all fit inside the free plan.