Skip to content
Carisly

CarislyBot

CarislyBot is the crawler of Carisly. It collects public job postings from employers: from their hiring systems (such as Recruitee, Greenhouse, Lever, Workable, Personio, Teamtailor, Ashby, Workday and SmartRecruiters) and from the structured job data (schema.org JobPosting) on their own website.

User-Agent: CarislyBot/1.0 (+https://carisly.com/bot)

What we collect and store

  • Only public job pages and job feeds; never pages behind a login.
  • Per job we only store: job title, employer, location, work type, salary (if the employer states it in structured form), date and the link to the job. Never the full job description.
  • On our site we always link to the job at the employer.

How often

Each employer once every 24 hours. The requests are spread over eight batches a day, so there are no peaks.

How we behave

  • robots.txt is leading (User-agent: CarislyBot or *), including Crawl-delay.
  • At most one request per second per domain, and a fixed limit on the total number of simultaneous requests.
  • On HTTP 429 or 503 we respect Retry-After and wait longer each time. If a site keeps refusing, we skip the domain for 7 days.
  • Where the site supports it, we only ask whether anything has changed (ETag / If-Modified-Since).
  • We never get around a block: no proxies, no rotating IP addresses, no other User-Agent, no logging in and no captchas.

Opting out

Don’t want CarislyBot to visit your site? Put this in your robots.txt:

User-agent: CarislyBot / Disallow: /

Or email us your domain or company name at contact@carisly.com. We process opt-outs within 24 hours: the crawler stops visiting and your jobs disappear from the site immediately.

Abuse or complaints

Is CarislyBot not behaving as described above, or do you want to report something? Email contact@carisly.com with your domain and the time; we reply within 24 hours.