Website Crawling

The Pitaj.io crawler enables your AI agent to discover, parse, and learn the complete content of your website. Simply enter your domain address, and the system automatically navigates subpages, articles, product descriptions, and contact sections.

---

How to Add a Website

1. Open the Knowledge Base Section
   From your agent’s dashboard menu, click Knowledge Base → Websites.

2. Select Input Mode
   Choose one of two ingestion modes:

   • Entire Domain: Enter e.g. https://yourcompany.com. The crawler detects your sitemap.xml or traverses internal links up to your depth limit.
   • Specific URLs: If you only want to index selected landing pages, documentation, or FAQ articles, enter individual URLs separated by commas or newlines.

3. Configure Exclusion Rules (Optional)
   In the Exclusion Patterns input, specify paths the crawler should ignore (e.g. /cart, /checkout, /admin, /login).

4. Initiate Crawling
   Click Start Crawling. The dashboard displays discovered pages and real-time training status indicators.

---

Sitemap Discovery & Handling

When you submit a domain, Pitaj.io automatically searches standard sitemap endpoints:

• https://yourdomain.com/sitemap.xml
• https://yourdomain.com/sitemap_index.xml
• https://yourdomain.com/wp-sitemap.xml (for WordPress sites)

Faster Large-Site Crawling

If your website contains over 500 subpages, directly entering the explicit URL to your product or content sitemap (e.g., https://yourdomain.com/sitemap-products.xml) yields the fastest and most complete ingestion.

---

Excluding Irrelevant URLs

To optimize knowledge storage and prevent the agent from citing non-content pages, define wildcard filter rules:

# Exclusion filter examples:
*/cart*
*/checkout*
*/my-account*
*/admin/*
*.pdf
*tag=*
*sort=*

---

Automated Re-indexing (Auto-Sync)

Business policies, catalog pricing, and schedules evolve over time. Configure auto-sync intervals in the crawler settings:

Interval	Best Suited For
Daily	E-commerce storefronts with frequent catalog and price updates
Weekly	Corporate websites, agencies, and SaaS marketing pages
Monthly	Static portfolios, documentation, and rarely updated pages
Manual	On-demand instant re-training with a single click on Sync Now

---

Troubleshooting & FAQ

Does the crawler support JavaScript (SPAs)?

Yes. Our ingestion engine utilizes headless Chromium rendering to accurately index dynamic single-page applications built in React, Vue, Angular, and Next.js.

Cloudflare or Bot Protection Issues?

If Cloudflare blocks the crawler with 403 or 503 errors, whitelist the Pitaj.io User-Agent or our outbound crawler IP addresses in your Cloudflare WAF Security Rules.