How to get list of User-Agent from access log
I wanted to block bots from accessing a website. For this, I need to generate a list of all browser User-Agent visiting the website, so I can see which ones can block. This work with most web servers like Apache, Nginx, IIS, etc.
To get the list of all User-Agents, run
|
1 |
awk -F\" '($2 ~ "^GET /"){print $6}' access_log | sort | uniq |
To get the List of all user agents with the number of visits, run
|
1 |
awk -F\" '($2 ~ "^GET /"){print $6}' access_log | sort | uniq -c | sort -n |
If you want to show the most visited User-Agents first, use “sort -nr” for reverse sorting.
|
1 |
awk -F\" '($2 ~ "^GET /"){print $6}' access_log | sort | uniq -c | sort -nr |
See Access Log