IP Whitelisting Guide
Whitelisting Requirements
To operate correctly, the Sensefuel platform needs to access your infrastructure for two purposes. Please whitelist the relevant IPs and/or user-agent on your firewall and server configuration.
| Purpose | Access required | IP/UA to whitelist |
|---|---|---|
| URL Crawling : Crawls your website to collect and reconcile product page URLs needed for performance dashboards & AI training | Your e-commerce website (HTTP/HTTPS) | User-Agent Sensefuel-bot and/or IP 34.38.112.126 and 34.141.247.168 |
| Feed & File Downloads : Downloads your product feeds | Your feed server (FTP / SFTP / HTTP / HTTPS) and port | 34.38.112.126 and 34.141.247.168 |
Security note: These IP addresses are exclusively dedicated to Sensefuel. Whitelisting them only allows Sensefuel-initiated outbound requests to reach your servers ; it does not open any inbound access from the public internet. Your existing authentication and access controls remain fully in effect.
Details
URL Crawling
Sensefuel crawls your website to identify and maintain an up-to-date mapping of your product page URLs. This is essential for reconciling URLs detected in customer tracking events (e.g. redirected or rewritten URLs) and for retrieving canonical URLs.
Even though your website is publicly accessible, whitelisting is recommended because automated crawl patterns (multiple sequential requests) may be flagged and blocked by your WAF, anti-bot protection, or rate-limiting rules.
What gets crawled
- Initial crawl: all URLs from your product catalog feed, plus any "unknown" URLs encountered in tracking events (redirects, rewrites, etc.).
- Subsequent crawls: only new or changed URLs are processed. If your URLs are stable, some crawl cycles may not request any page at all.
- Note : URLs present in your feed are crawled (once), in order to retrieve the canonical URL of each page.
Crawl behavior
Crawls are launched continuously throughout the day, approximately every 4 hours. Actual timing may vary slightly depending on other ongoing operations.
- Concurrency: up to 4 parallel requests at a time (the crawler waits for the 4 requests to be completed before launching the next one).
- Error handling: if your server returns repeated 5xx errors (500, 502, 503…), the crawl is automatically stopped after 10 consecutive errors to avoid overloading your infrastructure.
Feed & File Downloads
Sensefuel retrieves your product catalog feeds on a regular basis to keep your search index up to date. Downloads are performed over FTP, SFTP, or HTTP/HTTPS depending on your setup.
Make sure the whitelisted IPs above are allowed to connect to your feed server on the relevant port(s).