Knowledge bases
Website knowledge sources
A website source crawls a starting URL and turns successfully read pages into private, structured knowledge an AI Stand-in can retrieve. Crawl depth controls how far Stand follows same-site links, while plan allowances count pages that were actually indexed.
- Availability
- All plans
- Configured in
- Dashboard → AI Stand-ins → Knowledge bases → expand a knowledge base → Website sources → Crawl
- Category
- Knowledge bases
- Reference status
- Current
Create a website source
- Open or create a knowledge base, choose the website source type, and enter the starting URL.
- Choose a crawl depth from 0 through 10. Depth 0 indexes only the starting page; higher values permit additional same-site link levels.
- Start ingestion and monitor the source task until it succeeds, fails, or is cancelled.
- Attach the completed knowledge base to a Stand-in and save the Stand-in before expecting retrieval in Try it out or public chats.
Crawl boundaries and accounting
| Rule | Behavior |
|---|---|
| Same-site scope | Stand follows supported links that remain within the source website boundary rather than crawling arbitrary external domains. |
| Depth | Accepted values are 0 through 10 and limit link traversal from the starting page. |
| Per-crawl guardrail | One website crawl can attempt up to 1,000 pages. This is a platform guardrail, not an extra plan allowance. |
| Plan page usage | Only successfully indexed web pages count toward the organization’s advertised page allowance. |
| Indexed-content usage | Plans with a content allowance count extracted indexed text, not raw HTML, images, or retrieval vectors. |
What Stand preserves and removes
- Self-contained headings and their associated paragraphs.
- FAQ questions with their answers, nested lists, labeled table rows, and code examples with nearby explanation.
- Page URL, title, and section context used to ground retrieved facts.
- Relevant same-page glossary or parent context when a code example must be split for indexing.
- Repeated navigation, menus, and other page chrome are removed so they do not dominate retrieval.
Refresh and failure behavior
Every plan can manually refresh a website source it supports. Stand builds a replacement version while the prior successful content remains available to attached Stand-ins.
If the replacement succeeds, it atomically becomes current from the customer’s perspective. If crawling or indexing fails, the failure is shown while the prior successful content remains usable and temporary replacement work does not become permanent double-counted usage.
Plan availability
- Base includes one website knowledge base with up to 10 successfully indexed web pages.
- Pro includes website sources within its knowledge-base, 200-page, and 50 MB indexed-content allowances.
- Business includes website sources within its 200 MB indexed-content allowance and has no separately advertised page-count cap.
- Pricing remains the source of truth for current commercial allowances.
Example
The screen below shows the feature in its normal Stand context. Labels and surrounding controls may vary with account state and plan.
