About our crawler
Last updated: 2026-05-24
We catalog publicly listed Euro Truck Simulator 2 mods. To do that, our crawler visits mod listing pages, reads their metadata, and stores it in a database so we can verify facts and detect link rot. We do not download mod files, and we do not republish the original creator's full description.
Identification
| User-Agent | ModSentryBot/1.0 (+https://ets2modspot.com/bot) |
| Crawler operator | ETS2 Modspot |
| Hosting | Vercel + Hostinger |
| Contact | see site footer |
Source tiers
We classify every external source into one of five tiers that govern what we extract and how often we crawl. The same policy applies to every source in the same tier — we do not bend rules for any one site.
| Tier | Description | Rate |
|---|---|---|
| 1 | Partnered (written permission or feed) | as agreed |
| 2 | Public, robots.txt allows scraping | ≤ 6 req/min |
| 3 | Public, terms unclear — facts only, no verbatim | ≤ 3 req/min |
| 4 | Terms explicitly forbid — we do not crawl | 0 |
| 5 | Manual entry by an editor | — |
Rules we follow
- We honor
robots.txtstrictly — no exceptions. - We honor
Crawl-delaywhen present. - We respect HTTP
429and503by backing off exponentially. - We do not download mod archives (
.scs,.zip,.7z,.rar). - We do not re-crawl the same page within 24 hours unless an editor forces it.
- We do not fingerprint-evade. If a site blocks our named UA, we stop.
Opting out
If you operate a site we crawl and want us to stop, you have three options:
- Add a
Disallow: /for our User-Agent in yourrobots.txt. We will stop within one crawl cycle (≤ 24h). - Email the address in our site footer with the subject
opt-outand the domain. We will disable the source the same business day. - Block the User-Agent at your edge or WAF. We will detect the block and stop attempting requests.
For creators
If you are the creator of a mod we index, the better path is to claim it — that lets you direct traffic to your site, edit the description, and get a “Verified creator” badge on every mod page.