The solution for “touch: cannot touch `test.txt’: No space left on device”

In this guide, we are about to provide a simple solution for the titled issue “No space left on device”.

In case we receive “No space left on device” our first assumption will be mount point may get filled. Yeah, I go through the same but while looking into the specific mount point we found it has enough space left free. By following our thought should target “File System error” even filesystem have no error we may move to check for open files if that is not the case, at last, we can check the Inodes. Yes, here we have used with 100% of inodes under /var. finally, we have located thousands of files under /var/spool/clientmqueue.

We are trying to touch a file under /var and get “No space left on device”.

Step 1: Test by creating a file.

Let us try to create a new empty file under /var mount point. This may throw below error.

[root@ServerProd1 var]# touch test.txt
touch: cannot touch `test.txt': No space left on device
[root@ServerProd1 var]#

Step 2: Check space usage by listing mounted filesystem.

While checking the space under /var we have enough space.

[root@ServerProd1 ~]# df -h /var/
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VG01-LogVol03
4.7G 2.0G 2.5G 44% /var
[root@ServerProd1 ~]#

 

Step 3: Check filesystem for error.

By following we have checked the file state. Even it looks good.

# tune2fs -l /dev/mapper/VG01-LogVol03 | grep 'state'

[root@ServerProd1 ~]# tune2fs -l /dev/mapper/VG01-LogVol03 | grep 'state'
Filesystem state: clean
[root@ServerProd1 ~]#

 

Step 4: Check for Open files.

Check for the open files. Even we can check for the deleted open files.

# ulimit -n
# find /proc/*/fd -ls | grep '(deleted)'

 

At last, we are about to check the inodes.

Yes, here we found 100% of inodes are utilized.

 

Step 5: Check inode Usage for mounted filesystems.

Finally, let’s check for inode usage for the same filesystem. Here we found the issue because of filesystem running out of inode. To find the inode usage just add “i” with “df -h” command.

# df -hiP /var/

[root@ServerProd1 ~]# df -hiP /var/
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/VG01-LogVol03 313K 313K 0 100% /var
[root@ServerProd1 ~]#

 

Now we have found thousands of files which small in size under /var/spool/clientmqueue once we clear them we are in good stand for inode utilization under /var. Those files are just mail queue and safe to remove in our setup.

[root@ServerProd1 clientmqueue]# ls -lthr | wc -l
309821
[root@ServerProd1 clientmqueue]# pwd
/var/spool/clientmqueue

 

We have found the issue, once we found the number of files consumed under /var/spool/clientmqueue. That’s it,  here is the solution for “no space left on device” which caused by inode utilization.

Conclusion:

We may not get to know about inode usage until we configure in monitoring tools, In an Enterprise environment, it will be in place to monitor inode usages which will help to run applications without downtime. Hope this guide helps to a new Linux sysadmin to know the situation what blocks while creating a new file in Linux. Will come up with more troubleshooting articles in future. Subscribe to our newsletter and stay updated.