# Public HTML must be crawlable for its noindex directives to be seen. User-agent: * Allow: / Disallow: /api/ Disallow: /user_question_images/ Sitemap: https://www.oahelper.in/sitemap.xml User-agent: SemrushBot User-agent: AhrefsBot User-agent: DotBot User-agent: MJ12bot User-agent: PetalBot User-agent: Bytespider Disallow: /