#SemrushBot is run by a marketing service User-agent: SemrushBot Crawl-delay: 10 #serpstatbot is similar User-agent: serpstatbot Disallow: / #PetalBot is a Huawei search engine that is not important to us User-agent: PetalBot Disallow: / #Majestic SEO search seems to cause issues too User-agent: MJ12bot Disallow: / # https://opensiteexplorer.org/dotbot another SEO company User-agent: dotbot Disallow: / User-agent: DotBot/1.2 Disallow: / # https://yandex.com/support/webmaster/robot-workings/robot.html another SEO company User-agent: Yandex Disallow: / # https://seostar.co/robot/ another SEO company User-agent: adsbot Disallow: / # https://dataforseo.com/dataforseo-bot another SEO company User-agent: DataForSeoBot Disallow: / # https://seostar.co/robot/ another SEO company User-agent: DotBot/1.2 Disallow: /