{
  "id": 5717137,
  "title": "I built an AI API that turns any online store URL into a structured product catalog",
  "url": "https://urgent.news/2026/09/05/i-built-an-ai-api-that-turns-any-online-store-url-into-a-structured",
  "topic": "ai",
  "section": "AI",
  "published": "2026-09-05T04:03:08.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/crj_787403881c99662c653e8/i-built-an-ai-api-that-turns-any-online-store-url-into-a-structured-product-catalog-5775"
  },
  "original_language": "en",
  "account": "A savvy entrepreneur created an AI-powered API that transforms any online store's URL into a neatly organized product catalog. The API's functionality is simple: post a single URL, the store's homepage, and it will crawl the entire site, extracting essential details such as product name, brand, category, price, SKU, and photos. It also generates an AI-generated description for each product.\n\nThe process is straightforward: users only need to make a POST request to the API with the store's URL as the input. The response returns a job ID, which can be used to track the progress of the extraction. The AI doesn't require any configuration for CSS selectors or per-site templates, as it reads each page and understands the content naturally.\n\nHowever, the process isn't without challenges. E-commerce pages often mislead the crawler: category pages can appear identical to product pages, and the same product may be listed on different URLs with varying descriptions. To address these issues, the API includes three cleanup guards: a listing-card detector, a deduplication pass with slug-digit tie-breaking, and a parent-category pruner.\n\nThe API's performance was tested using a Brazilian industrial-tools store. In a cold start scenario, the API crawled 160 pages, produced 126 candidates, and generated 50 clean products in just 72 seconds, with no duplicates or category pages included in the results. The API respects the store's robots.txt by default but offers an owner-override flag for exceptions.\n\nWhile the API is currently compatible with non-JavaScript-rendered storefronts, future plans include support for dynamic storefronts. The API also assumes one page equals one product, which may not be applicable to all stores, particularly those with filterable listings or unique SKUs.\n\nThe API is accessible through a free tier on RapidAPI, offering 50 calls per month without the need for a credit card. A live demo is available, showcasing a real extraction of 50 products complete with photos. The entrepreneur encourages users to share any edge cases they encounter, as these insights will help shape the API's roadmap.",
  "summary": "I watched a store owner retype 600 products by hand while migrating platforms. It took her team two weeks. That pain became this API. What it does You POST one URL — the homepage of any online store. The API crawls it the way a person would (categories, pagination, product pages) and returns the entire catalog as structured JSON: name, brand, category, price, SKU, photos, and an AI-written…",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}