pyinotify yelling no space available

[Mattia Procopio]

don't know why, but in the last days whenever I try to run manage.py runserver I got a pyinotify error, while I'm still able to surf to my pages and work with them I see a nice traceback on console, what's going on?

Traceback:

September 19, 2014 - 10:04:00
Django version 1.7, using settings 'uwn.settings.local'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.
[2014-09-19 10:04:00,066 pyinotify ERROR] add_watch: cannot watch /usr/lib64/python2.7/site-packages/numpy/core/_methods.py WD=-1, Errno=No space left on device (ENOSPC)
[2014-09-19 10:04:00,067 pyinotify ERROR] add_watch: cannot watch /usr/lib/python2.7/site-packages/rest_framework/permissions.py WD=-1, Errno=No space left on device (ENOSPC)
[2014-09-19 10:04:00,067 pyinotify ERROR] add_watch: cannot watch /usr/lib64/python2.7/site-packages/django/utils/termcolors.py WD=-1, Errno=No space left on device (ENOSPC)
[2014-09-19 10:04:00,067 pyinotify ERROR] add_watch: cannot watch /usr/lib64/python2.7/site-packages/django/shortcuts.py WD=-1, Errno=No space left on device (ENOSPC)
[2014-09-19 10:04:00,067 pyinotify ERROR] add_watch: cannot watch /usr/lib64/python2.7/shutil.py WD=-1, Errno=No space left on device (ENOSPC)
[2014-09-19 10:04:00,067 pyinotify ERROR] add_watch: cannot watch /usr/lib64/python2.7/lib-dynload/_heapq.so WD=-1, Errno=No space left on device (ENOSPC)
[2014-09-19 10:04:00,067 pyinotify ERROR] add_watch: cannot watch /usr/lib64/python2.7/lib-dynload/_randommodule.so WD=-1, Errno=No space left on device (ENOSPC)
[2014-09-19 10:04:00,067 pyinotify ERROR] add_watch: cannot watch /usr/lib64/python2.7/site-packages/numpy/fft/fftpack.py WD=-1, Errno=No space left on device (ENOSPC)
[remaining messages cut]

[Baptiste Mispelon]

Sorry if that's obvious, but have you checked that you still have space left on your root partition (or wherever /usr/lib64 is mounted)?

[Mattia Procopio]

yep, I have just 23% used

➜  django git:(django-support-testing) df /usr/lib64 
Filesystem              1K-blocks     Used Available Use% Mounted on
/dev/mapper/fedora-root  51475068 10943904  37893340  23% /

[Baptiste Mispelon]

And what does df -i report (​http://serverfault.com/questions/427091/no-space-left-on-device-but-there-is-space#427092)?

[Mattia Procopio]

➜  patients_database git:(big_field) df -i
Filesystem                  Inodes  IUsed      IFree IUse% Mounted on
/dev/mapper/fedora-root    3276800 354869    2921931   11% /
devtmpfs                   2039988    511    2039477    1% /dev
tmpfs                      2042567     11    2042556    1% /dev/shm
tmpfs                      2042567    694    2041873    1% /run
tmpfs                      2042567     15    2042552    1% /sys/fs/cgroup
tmpfs                      2042567     30    2042537    1% /tmp
/dev/mapper/fedora-home   11395072 238864   11156208    3% /home
/dev/sda5                   128016     83     127933    1% /boot
/dev/sda4               4294967295   2886 4294964409    1% /boot/efi

[Baptiste Mispelon]

What about sysctl -n fs.inotify.max_user_watches? (​https://github.com/seb-m/pyinotify/wiki/Frequently-Asked-Questions)

[Claude Paroz]

Shortened the description, we don't need all messages...

[Mattia Procopio]

yes, I can increase to a bigger number. But this is not a real solution to the problem. I just want to understand if this is a django problem or not. If I try to run into a virtual env I don't get the error

[Aymeric Augustin]

We're also trying to understand if this is a problem with Django or with your system.

Considering that it works in a virtualenv and that we haven't pinpointed the root cause, I'm going to close the ticket as "needsinfo". Please reopen if you understand why you're getting these messages. Then we could attempt to provide at least a better error message.

[alexandervlpl]

I'm suddenly having the exact same problem with a Django 1.8 project that was running perfectly a few days ago. Moreover, starting manage.py runserver in a different project on the same system does not give me these errors.

For the record I have one partition, 64% used. I even wrote a large file to it with dd to be absolutely sure there is space on the device. Disk is healthy.

EDIT: Yes, increasing max_user_watches as mentioned above does resolve this:

sudo sysctl -n -w fs.inotify.max_user_watches=16384

Maybe a note should be made about this in the Django docs?

[Mattia Procopio]

which OS you use?

[alexandervlpl]

OS: Ubuntu 14.04 LTS. Updated my comment (increasing max_user_watches does resolve this).

[Mattia Procopio]

I tried but after couple of days I get the same, let me know how it will go

[Aman Verma]

Hello, did anyone figure out the source of this issue ? I do not have sudo access on my machine, I can't increase number of watches. Is their an alternative solution ?

[sverker wahlin]

Having the same problem on a fresh Fedora 26 installation. I have roughly 250 files in my entire devserver directory.
I do, however, have a couple of symbolic links. Not sure if that's causing any form of system panic/nervousness.
I have 8192 as max_user_watches.
I also have a nextcloud installation running. One of those folders contain over 30 000 files. I'm not sure if that has added inotify signals for all of the files, but it is a possible source.

Upping the max_user_watches to 16384 fixed it, for now. Will see if it creeps up again.