How to block bad bots User-Agents in Cloudflare
Cloudflare provides a way to block bad bots under
Domain > Security > Bots.
If you want to block one bots that is allowed, you can block them by User Agent. To see what bots visit your website, see How to get list of User-Agent from access log. This will get list of all bots accessing your website from Apache, Nginx or similar webserver access log.
You can block visitors by User-Agent by going to
Cloudflare > Domain > Security > WAF
Click on “Create firewall rule”.
On next page select following values
Field = User Agent
Operator = contains
Value = Enter the user agent you need to block
On this page, you can also add rules by clicking on “Edit expression” link, and then pasting the rule. I used the following rule to block marketing/SEO bots.
(http.user_agent contains "semrush") or (http.user_agent contains "ahrefs") or (http.user_agent contains "BLEXBot") or (http.user_agent contains "mj12bot") or (http.user_agent contains "opensiteexplorer") or (http.user_agent contains "megaindex") or (http.user_agent contains "dataforseo") or (http.user_agent contains "petalsearch") or (http.user_agent contains "Barkrowler") or (http.user_agent contains "Go-http-client") or (http.user_agent contains "Mechanize") or (http.user_agent contains "Mechanize") or (http.user_agent contains "woorank") or (http.user_agent contains "spyfu") or (http.user_agent contains "majestic")
NOTE: Don’t use this rule without knowing what it does, you may not want to block all of these bots. Do your own research on what the bot does before blocking it from your website.
Related Posts
How to get list of User-Agent from access log
How to block Bad Bots (User Agents) using .htaccess
Back to Cloudflare