CTP AI: the crawler behind the quotations

How it identifies itself, what it reads from a vendor's pages, how to allow or refuse it in robots.txt, how a claim is verified, and the feed a vendor can publish so its prices are read from its own hand instead of its pages.

Identity

User agent: CTP-AI/1.0 (+https://comparethepeptide.org/ctp-ai/). It comes from this site's own server, 159.195.70.43, and from nowhere else. It reads robots.txt first, with that agent, and obeys both the CTP-AI rules and the * rules. It asks for at most one page a second per host, follows the sitemaps a site publishes (sitemap.xml, wp-sitemap.xml, sitemap_index.xml) to the product pages, and reads the structured product data on them. Pages with no structured product data are skipped and counted on the public report.

Allow it, or refuse it, the way you do for Googlebot

User-agent: CTP-AI
Allow: /

# to refuse it:
User-agent: CTP-AI
Disallow: /

A site that refuses it is not crawled and not quoted; its claim can still be verified by the token file, and its feed is still read, because the vendor registered that address itself.

What it reads from a product page

JSON-LD Product (schema.org) in the page: name, and the offers with price, priceCurrency, availability, url.
From the name: the peptide (matched against the registry, including vendor spellings such as TB500) and the milligrams (e.g. "10mg", "5 x 10 mg"). A layered product that names several peptides is quoted under each, with the total quantity, marked as layered.
The format from the name: vial, dual-chamber pen, pre-mixed liquid, Lyoprester kit. Without a word for it, the vendor's default (lyophilised vial).
Consumables by name: bacteriostatic water, insulin syringes, pen needles; shown as places to buy beside the reference price.

Every run writes report.json: per vendor, pages read, products found, pages without structured data, products naming no registry peptide. Prices are gathered every four hours; claims and feeds hourly; vendor discovery daily. Every file carries its gathered_at.

How vendors are found

Three ways, all public. The seed list. A claim, once verified. And discovery: every registry peptide name is searched, for every country in rotation, every day; each domain the search returns is probed for structured product data that names a registry peptide and carries a price, and the result, found or not, is published with its evidence in discovered.json. A domain that proves to be a vendor is crawled from the next run on.

Verifying a claim

A claim issues a token. Publish it at https://<your domain>/.well-known/ctp-verify.txt or https://<your domain>/ctp-verify.txt (a plain text file holding the token; the site root is for shops whose server hides dot-directories) or as <meta name="ctp-verify" content="<token>"> on your home page. The crawler checks it within the hour; the public status is on panel.json and on your vendor page.

The feed

A verified vendor may register one JSON address on its own domain. It is read every hour and replaces that vendor's crawled products: the prices are then the vendor's own hand, dated to the hour. Top level: vendor, country, currency, generated, products. Each product:

Field		Meaning
`name`	required	the product name; the peptide name and the milligrams are read from it when the fields below are absent
`url`	required	the product page (https); the click-out goes there with ?ref=comparethepeptide&ctp=<id> appended
`price`	required	a number: the price a visitor pays
`currency`	required	ISO 4217, e.g. GBP; the feed's top-level currency is the default
`quantity_mg`	optional	milligrams of peptide in the product (the total for a layered product)
`peptides`	optional	registry slugs, e.g. ["bpc-157", "tb-500"]; otherwise matched from the name
`format`	optional	one of vial_lyophilised, pen_dual_chamber, liquid_prefilled, lyoprester_kit; otherwise read from the name
`availability`	optional	InStock, OutOfStock or PreOrder
`test_report_url`	optional	the batch's test report (COA), https; shown as a link beside the product
`batch`	optional	the batch the test report belongs to
`shelf_life_days_after_opening`	optional	the vendor's own in-use figure for this product; replaces the 28-day convention, labelled as the vendor's statement
`ships_to`	optional	ISO country codes the product ships to; it is quoted in each of them (default: the feed's country)

Example:

{
 "vendor": "Example Peptides Ltd",
 "country": "GB",
 "currency": "GBP",
 "generated": "2026-10-09",
 "products": [
  {
   "name": "BPC-157 10 mg",
   "url": "https://example.co.uk/product/bpc-157-10mg/",
   "price": 29.5,
   "currency": "GBP",
   "quantity_mg": 10,
   "peptides": [
    "bpc-157"
   ],
   "format": "vial_lyophilised",
   "availability": "InStock",
   "test_report_url": "https://example.co.uk/coa/bpc-157-batch-2410.pdf",
   "batch": "2410",
   "ships_to": [
    "GB",
    "IE"
   ]
  }
 ]
}