Tag: log

How to get list of User-Agent from access log
I wanted to block bots from accessing a website. For this, I need to generate a list of all browser User-Agent visiting the website, so I can see which ones can block. This work with most web servers like Apache, Nginx, IIS, etc.

To get the list of all User-Agents, run
```
awk -F\" '($2 ~ "^GET /"){print $6}' access_log | sort | uniq
```
To get the List of all user agents with the number of visits, run
```
awk -F\" '($2 ~ "^GET /"){print $6}' access_log | sort | uniq -c | sort -n
```
If you want to show the most visited User-Agents first, use “sort -nr” for reverse sorting.
```
awk -F\" '($2 ~ "^GET /"){print $6}' access_log | sort | uniq -c | sort -nr
```
See Access Log
October 29, 2022
Display real time statistics with Logtop
Logtop is a real-time log analysis tool. It can be used to understand log files. The developer describes it as “Display real time statistics of whatever you want.”. You can pass any value to it, logtop aggregate the data and show it by the number of times the data appear. Most repeated items shows on top of the list.

To install logtop on Ubuntu/Debian, run
```
apt install logtop -y
```
To see all IP address that is accessing your web server, run
```
tail -f access_log | awk {'print $1; fflush();'} | logtop
```
To see the web page that gets the most requests, run
```
tail -f access_log |  awk {'print $7; fflush();'} | logtop
```
See log
June 10, 2021