SpeakDuoBot
SpeakDuoBot is the crawler behind the SpeakDuo tutor directory, which helps language learners compare tutors and prices across online tutoring marketplaces. If you saw it in your logs, this page explains what it does.
User agent
SpeakDuoBot always identifies itself with this user agent and never pretends to be a browser:
SpeakDuoBot/1.0 (+https://www.speakduo.com/bot)What it collects
Only short, public facts from public tutor profile and listing pages (and the sitemaps that list them): display name, profile photo URL, languages taught and spoken, country, lesson and trial price, rating, review count and specialties.
We don't copy bios, reviews or messages, we never log in, and we don't call private APIs. Every tutor in the directory links back to the marketplace, where the booking happens.
robots.txt
SpeakDuoBot follows the Robots Exclusion Protocol (RFC 9309), including Disallow and Crawl-delay. If a site's robots.txt can't be fetched, it doesn't crawl that site. To block it completely, add:
User-agent: SpeakDuoBot
Disallow: /Pages that become disallowed are dropped from the directory on the next crawl.
Crawl rate
- One request at a time per site, at least 3 seconds apart (longer if your Crawl-delay asks for it).
- Listing pages are refreshed at most daily and profiles at most weekly.
- A 403 or a bot challenge stops the crawl of that site for the run. We never solve captchas or rotate IP addresses.
- Profiles that return 404/410 are removed from the directory.
Opt out or request removal
Tutors and platforms can ask us to remove a profile or stop crawling a site at any time. Email support@speakduo.com with the profile URL (or your domain) and we'll take it down from the directory.