Articles in this section

Editorial content crawler

  • Updated

1. Introduction

Sensefuel Max enables the automatic integration of editorial content within search results to inspire, guide and reassure customers throughout their purchasing journey.

What is editorial content?
It refers to any page from your website or blog that contains at least:

  • A title
  • An image
  • A description

Typical examples include:

  • Recipes
  • Buying guides
  • Tutorials
  • Product comparisons
  • News
  • Blog posts
  • Store / shop information

Examples from clients:

Editorial content helps create inspiration and trust, significantly contributing to an improved search and navigation experience.

2. What Type of Content Can Be Crawled?

✅ Usable / Crawlable Content

  • Buying guides
  • Recipes
  • Tutorials
  • Articles / News
  • Product comparisons
  • Blog posts
  • Store / Local shop information

❌ Not usable as editorial content

  • Banners
  • Ads
  • Product descriptions
  • FAQ
  • Standalone images
  • Any element without semantic editorial purpose

3. How Sensefuel Retrieves Editorial Content

Sensefuel uses a two-step process:

Step 1 — Retrieve the list of URLs to crawl

Two methods are possible:

✅ 1. XML sitemaps (recommended)

  • Lowest development cost
  • Stable and scalable
  • Highly reliable

✅ 2. HTML page crawling (DOM-based)

Sensefuel can analyze one or more HTML pages containing lists of editorial content links and extract them via HTML tags.

Example :
Extraction of all <a> tags inside <article class="PostCard">.

⚠ HTML structures change frequently → this method may require maintenance.

Both options must be discussed with your Customer Success Manager to determine the best approach and any additional rules.

Step 2 — Crawl each editorial content page

Sensefuel collects structured and unstructured data from the page to understand, classify, and index editorial content.

4. Structured Data Crawling (recommended)

Sensefuel can rely on multiple structured data types.
The recommended method is JSON-LD, as it is the most robust and semantic-friendly format.

Minimum JSON-LD fields required

Property Type Description
type text Content type
name text Content title
description text Content description
image URL Main image

Additional properties

Depending on the content typology (recipe, guide, tutorial, shop…), additional fields from schema.org can be collected to enrich the content, improve classification, and generate filters.

Schema.org examples:

  • Recipe → https://schema.org/Recipe
  • Guide → https://schema.org/Guide
  • HowTo → https://schema.org/HowTo
  • LocalBusiness / Shop → https://schema.org/LocalBusiness

5. Other Types of Structured Data Supported

If JSON-LD is absent, incomplete, or requires completion, Sensefuel can also read:

Microdata

Using HTML attributes such as:

  • itemprop
  • itemtype

RDFa

Using HTML attributes such as:

  • property
  • typeof

HTML content extraction

Sensefuel may also extract specific HTML elements to enhance understanding or classification, although this method is less recommended due to fragility when HTML changes.

6. Using Editorial Content in Sensefuel

Once crawled, processed, enriched, and classified, editorial contents become available in:

Search Layer (JavaScript Tag)

Contents appear when relevant based on the user’s query.

Discovery API

They can be displayed in any context of your e‑commerce experience:

  • Search suggestion
  • Search results page
  • Homepage
  • Product listing page
  • Product detail page
  • Category pages
  • And more…

This allows editorial content to support the commercial strategy at every step.

7. Benefits of Integrating Editorial Content

  • Inspires customers early in their journey
  • Reassures and guides decision‑making
  • Provides a richer and more engaging user experience

Related articles