[Bots] Managed robots.txt

src/content/docs/bots/additional-configurations/managed-robots-txt.mdx

@@ -0,0 +1,36 @@
+---
+pcx_content_type: reference
+title: Direct AI crawlers with managed robots.txt
+sidebar:
+  order: 8
+  label: Managed robots.txt
+---
+
+import { Render, Tabs, TabItem } from "~/components"
+
+Protect your website or application from AI crawlers by implementing a `robots.txt` file on your domain to direct AI bot operators on what content they can and cannot scrape for AI model training. 
+
+Cloudflare's managed `robots.txt` explicitly disallows known bots engaged in scraping for AI purposes and AI agent activity. 
+
+AI bots are expected to follow the `robots.txt` directives. Otherwise, they risk getting banned. 
+
+To implement a `robots.txt` file on your domain based on your plan:
+
+<Tabs>
+  <TabItem label="Bot Fight Mode">
+    <Render file="enable-managed-robots-txt" params={{ one: "Bot Fight Mode" }} />
+  </TabItem>
+  <TabItem label="Super Bot Fight Mode">
+    <Render file="enable-managed-robots-txt" params={{ one: "Super Bot Fight Mode" }} />
+  </TabItem>
+    <TabItem label="Bot Management for Enterprise">
+    <Render file="enable-managed-robots-txt" params={{ one: "Bot Management" }} />
+  </TabItem>
+</Tabs>
+
+:::note
+When you enable the `robots.txt` file on your domain, existing `robots.txt` files will be replaced.
+:::
+
+## Availability
+Managed `robots.txt` for AI crawlers is available on all plans.

src/content/docs/bots/additional-configurations/static-resources.mdx

@@ -2,7 +2,7 @@
 pcx_content_type: reference
 title: Static resource protection
 sidebar:
-  order: 8
+  order: 9
 
 ---
 

src/content/docs/security/settings.mdx

@@ -100,15 +100,16 @@ This section allows you to configure multiple security-related settings. The fol
 | [Custom username and password location](/waf/detections/leaked-credentials/#custom-detection-locations)                 | **Security** > **Settings**                                                                                         |
 | [Custom content location](/waf/detections/malicious-uploads/#custom-scan-expressions)                                   | **Security** > **Settings**                                                                                         |
 | [Custom sensitive data deployment](/waf/managed-rules/reference/sensitive-data-detection/#configure-in-the-dashboard)   | **Security** > **Sensitive Data**                                                                                   |
-| [Block definitely automated traffic](/bots/get-started/super-bot-fight-mode/#bot-settings)                                       | **Security** > **Bots** > **Configure Super Bot Fight Mode<br/>Security** > **Bots** > **Configure Bot Management** |
-| [Block likely bots](/bots/get-started/super-bot-fight-mode/#bot-settings)                                                        | **Security** > **Bots** > **Configure Super Bot Fight Mode<br/>Security** > **Bots** > **Configure Bot Management** |
-| [Allow verified bots](/bots/get-started/super-bot-fight-mode/#bot-settings)                                                      | **Security** > **Bots** > **Configure Super Bot Fight Mode<br/>Security** > **Bots** > **Configure Bot Management** |
-| [Static resource protection](/bots/additional-configurations/static-resources/)                                                         | **Security** > **Bots** > **Configure Super Bot Fight Mode<br/>Security** > **Bots** > **Configure Bot Management** |
-| [Optimize for WordPress](/bots/troubleshooting/wordpress-loopback-issue/)                                                     | **Security** > **Bots** > **Configure Super Bot Fight Mode<br/>Security** > **Bots** > **Configure Bot Management** |
-| [JavaScript detections](/bots/additional-configurations/javascript-detections/)                                                         | **Security** > **Bots** > **Configure Super Bot Fight Mode<br/>Security** > **Bots** > **Configure Bot Management** |
+| [Block definitely automated traffic](/bots/get-started/super-bot-fight-mode/)                              | **Security** > **Bots** > **Configure Super Bot Fight Mode<br/>Security** > **Bots** > **Configure Bot Management** |
+| [Block likely bots](/bots/get-started/super-bot-fight-mode/)                                               | **Security** > **Bots** > **Configure Super Bot Fight Mode<br/>Security** > **Bots** > **Configure Bot Management** |
+| [Managed `robots.txt`](/bots/additional-configurations/managed-robots-txt/)                                             | **Security** > **Bots** > **Configure Bot Fight Mode<br/>Security** > **Bots** > **Configure Super Bot Fight Mode<br/>Security** > **Bots** > **Configure Bot Management** |
+| [Allow verified bots](/bots/get-started/super-bot-fight-mode/)                                             | **Security** > **Bots** > **Configure Super Bot Fight Mode<br/>Security** > **Bots** > **Configure Bot Management** |
+| [Static resource protection](/bots/additional-configurations/static-resources/)                                         | **Security** > **Bots** > **Configure Super Bot Fight Mode<br/>Security** > **Bots** > **Configure Bot Management** |
+| [Optimize for WordPress](/bots/troubleshooting/wordpress-loopback-issue/)                                               | **Security** > **Bots** > **Configure Super Bot Fight Mode<br/>Security** > **Bots** > **Configure Bot Management** |
+| [JavaScript detections](/bots/additional-configurations/javascript-detections/)                                         | **Security** > **Bots** > **Configure Super Bot Fight Mode<br/>Security** > **Bots** > **Configure Bot Management** |
 | [Auto-update machine learning model](/bots/reference/machine-learning-models/)                                          | **Security** > **Bots** > **Configure Bot Management**                                                              |
 | [Enable Security.txt](/security-center/infrastructure/security-file/)                                                   | **Security** > **Settings**                                                                                         |
-| [Challenge Passage](/fundamentals/security/cloudflare-challenges/challenge-passage/)                                                                      | **Security** > **Settings**                                                                                         |
+| [Challenge Passage](/fundamentals/security/cloudflare-challenges/challenge-passage/)                                    | **Security** > **Settings**                                                                                         |
 | [Browser Integrity Check](/waf/tools/browser-integrity-check/)                                                          | **Security** > **Settings**                                                                                         |
 | [Replace insecure JavaScript libraries](/waf/tools/replace-insecure-js-libraries/)                                      | **Security** > **Settings**                                                                                         |
 | [Security Level](/waf/tools/security-level/)                                                                            | **Security** > **Settings**                                                                                         |

src/content/partials/bots/enable-managed-robots-txt.mdx

@@ -0,0 +1,11 @@
+---
+inputParameters: params1
+
+---
+
+import { Markdown } from "~/components"
+
+1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/) and select your account and domain.
+2. Go to **Security** > **Bots**.
+3. Select **Configure {props.one}**.
+4. On **Manage bot traffic with robots.txt**, select the toggle.

src/content/release-notes/bots.yaml

@@ -5,6 +5,10 @@ productLink: "/bots/"
 productArea: Application security
 productAreaLink: /fundamentals/reference/changelog/security/
 entries:
+  - publish_date: "2025-04-28"
+    title: Managed robots.txt is now available
+    description: |-
+      Direct AI crawlers on what they can and cannot scrape from your website or application by [implementing a `robots.txt` file](/bots/additional-configurations/managed-robots-txt/) to your domain.
   - publish_date: "2025-04-24"
     title: Bot Detection Alerts are now available
     description: |-