Articles in this section

See more

How to identify duplicated IDs/offers in your feed ?

  • Updated

You have received an error message indicating that we were unable to compile your feed because duplicate IDs were detected. This article explains how to quickly identify duplicates in your file.

1. If Your Feed Is a CSV File

Open a new Excel spreadsheet and import your CSV file:

  1. Go to the Data tab.
  2. Select From Text/CSV.
  3. Choose your CSV file and import it into Excel.

Once your data is loaded:

  1. Select the ID column.
  2. Go to the Home tab and click Conditional Formatting.
  3. Select Highlight Cells Rules > Duplicate Values.
  4. Choose a highlight color and click OK.

Excel will automatically highlight all duplicate IDs.

To make them easier to review:

  1. Apply a filter to the column.
  2. Sort or filter by color.
  3. Review the highlighted rows to identify duplicated IDs.

2. If Your Feed Is an XML File

First, convert your XML file into an Excel worksheet:

  1. Enable the Developer tab in Excel if it is not already visible.
  2. Go to Developer > Import.
  3. Select your XML file and follow the import wizard.

Once the data has been imported, follow the same steps described in the CSV File section above to highlight duplicate IDs.

Note: For very large files, Excel may reach its processing limits. In such cases, you can try the same procedure using LibreOffice Calc or extract only the ID column into a smaller spreadsheet before performing the duplicate check.

3. If Your Feed Is an Offer Feed (Uniqueness Based on id + store_code)

For offer feeds, uniqueness is not determined by the id field alone. Instead, each offer is uniquely identified by the combination of:

  • id
  • store_code

This means that the same id can legitimately appear multiple times as long as it is associated with different stores.

Step 1: Create a Combined Key

Add a new helper column to your spreadsheet.

For example:

  • Column A = id
  • Column B = store_code

In a new column, enter the following formula:

=A2&"_"&B2

This creates a combined key composed of both values.

Example:

idstore_codeCombined Key
12345PARIS12345_PARIS
12345LILLE12345_LILLE
12345PARIS12345_PARIS

In this example:

  • 12345_PARIS appears twice and is therefore a duplicate.
  • 12345_LILLE is not a duplicate because the store code is different.

Step 2: Highlight Duplicate Combinations

Apply conditional formatting to the Combined Key column:

  1. Select the entire Combined Key column.
  2. Go to Home > Conditional Formatting.
  3. Select Highlight Cells Rules > Duplicate Values.
  4. Choose a highlight color and click OK.

Excel will highlight all duplicated id + store_code combinations.

Step 3: Filter by Color

To quickly locate the problematic rows:

  1. Apply a filter to the Combined Key column.
  2. Sort or filter by the highlight color.
  3. Review the duplicated combinations.

Alternative Method: Pivot Table

For very large offer feeds, you can create a Pivot Table using the Combined Key column:

  1. Select your dataset.
  2. Go to Insert > PivotTable.
  3. Add the Combined Key field to both Rows and Values.
  4. Configure the Values field to display a Count.

Any combination with a count greater than 1 represents a duplicate offer.

Additional Notes for Large Files

Excel may reach its processing limits when working with very large feed files.

If this happens, you can:

  • Try the same procedure using LibreOffice Calc, which can sometimes handle large datasets more efficiently.
  • Extract only the relevant columns into a separate spreadsheet:
    • the ID column for product feeds;
    • the ID and store_code columns for offer feeds.
  • Use your preferred AI assistant (such as Gemini, ChatGPT, Microsoft Copilot, etc.) to analyze the extracted data and identify duplicate entries.

When using an AI assistant, make sure to remove any sensitive or confidential information from your file before uploading it.