How to block bad bots User-Agents in Cloudflare

Cloudflare provides a way to block bad bots under

Domain > Security > Bots.

If you want to block one bots that is allowed, you can block them by User Agent. To see what bots visit your website, see How to get list of User-Agent from access log. This will get list of all bots accessing your website from Apache, Nginx or similar webserver access log.

You can block visitors by User-Agent by going to

Cloudflare > Domain > Security > WAF

Click on “Create firewall rule”.

On next page select following values

Field = User Agent
Operator = contains
Value = Enter the user agent you need to block

On this page, you can also add rules by clicking on “Edit expression” link, and then pasting the rule. I used the following rule to block marketing/SEO bots.

(http.user_agent contains "semrush") or (http.user_agent contains "ahrefs") or (http.user_agent contains "BLEXBot") or (http.user_agent contains "mj12bot") or (http.user_agent contains "opensiteexplorer") or (http.user_agent contains "megaindex") or (http.user_agent contains "dataforseo") or (http.user_agent contains "petalsearch") or (http.user_agent contains "Barkrowler") or (http.user_agent contains "Go-http-client") or (http.user_agent contains "Mechanize") or (http.user_agent contains "Mechanize") or  (http.user_agent contains "woorank") or (http.user_agent contains "spyfu") or (http.user_agent contains "majestic")

NOTE: Don’t use this rule without knowing what it does, you may not want to block all of these bots. Do your own research on what the bot does before blocking it from your website.

Related Posts

How to get list of User-Agent from access log
How to block Bad Bots (User Agents) using .htaccess

Back to Cloudflare