No space left on device

On a server, when creating new files, i get error – “No space left on device”.

root@sok:~# touch 2
touch: cannot touch `2': No space left on device
root@sok:~#

Check disk space with df -h, it shows 129 GB disk space is free

root@sok:~# df -h
Filesystem                                              Size  Used Avail Use% Mounted on
rootfs                                                  383G  243G  129G  66% /
udev                                                     10M     0   10M   0% /dev
tmpfs                                                   1.6G  224K  1.6G   1% /run
tmpfs                                                   5.0M     0  5.0M   0% /run/lock
tmpfs                                                   3.5G     0  3.5G   0% /run/shm
tmpfs                                                   3.5G  4.0K  3.5G   1% /tmp
root@sok:~#

This can be due to inodes getting full, check inodes usage with “df -i”.

root@sok:~# df -i
Filesystem                                               Inodes    IUsed   IFree IUse% Mounted on
rootfs                                                 25419184 25419184       0  100% /
udev                                                    2057538      319 2057219    1% /dev
tmpfs                                                   2059026      258 2058768    1% /run
tmpfs                                                   2059026        3 2059023    1% /run/lock
tmpfs                                                   2059026        2 2059024    1% /run/shm
tmpfs                                                   2059026        7 2059019    1% /tmp
root@sok:~#

As you can use Inodes are all used up.

To find where inodes are used up, run

find / -xdev -printf '%h\n' | sort | uniq -c | sort -k 1 -n

If you have recent version of du, you can use

du --inodes -xS | sort -n

This will display all folders with inode usage, sorted by inode usage.