OilOnCottonBot

OilOnCottonBot is the crawler behind Oil on Cotton, an independent, non-commercial index of clothing sold by museum shops. It is not affiliated with any museum or with Shopify.

What it reads

Once a week it reads each listed shop's robots.txt, its clothing collection pages, and the product pages those collections link to. It keeps the product name, price, sizes in stock and the page address. It does not keep descriptions or images, and it calls no shop or platform API.

How it behaves

It sends one request at a time, at least one second apart per shop, follows Crawl-delay and Retry-After, and stops for the week if a shop keeps answering 429. Product pages it has found not to be clothing are rechecked once a month, not every week.

Every request carries the user agent OilOnCottonBot and a Web Bot Auth signature (Ed25519). The public key is published at /.well-known/http-message-signatures-directory.

Blocking it

Add this to your robots.txt and the next visit will stop there:

User-agent: OilOnCottonBot
Disallow: /

Contact

Source code and crawler rules: github.com/yusufaf/oil-on-cotton. To have a shop or a listing removed, open an issue there. Removals are made within 48 hours.