How to get list of User-Agent from access log

I wanted to block bots from accessing a website. For this, I need to generate a list of all browser User-Agent visiting the website, so I can see which ones can block. This work with most web servers like Apache, Nginx, IIS, etc.

To get the list of all User-Agents, run

awk -F\" '($2 ~ "^GET /"){print $6}' access_log | sort | uniq

To get the List of all user agents with the number of visits, run

awk -F\" '($2 ~ "^GET /"){print $6}' access_log | sort | uniq -c | sort -n

If you want to show the most visited User-Agents first, use “sort -nr” for reverse sorting.

awk -F\" '($2 ~ "^GET /"){print $6}' access_log | sort | uniq -c | sort -nr

See Access Log