Use pyinotify (where available) instead of polling filesystem every second to detect changes

[Chris Lamb]

This patch series adds support for pyinotify to replace the "poll-every-one-second" mechanism in django.utils.autoreload on systems where that library is available.

Inotify is an event-driven notifier for monitoring filesytems changes. Instead of dancing around the filesystem comparing modification times every second, we can simply register all the files we are interested in and let the thread block until one of them changes. When this happens, we reload the server as before.

This has the following advantages over the current polling system:

• Scales far better to larger projects.
• Response time to code changes is drastically reduced - the server restarts pretty much instantly instead of having to wait ~0.5 seconds for the reload.
• More robust change detection - pretty much all forms of modification of the interested file are noticed (in particular, moving the file and editing the file without changing the mtime)
• Saves battery usage - I have to use --noreload on my laptop (with encrypted filesystems) as it noticably reduces my battery life. Django is killing kittens.
• Cleaner implementation - Polling is just dirty, and requires us to maintain a huge dict full of modification times.

Whilst inotify is Linux-specific, the patch gracefully degrades if a suitable version of pyinotify is not available. It does not add pyinotify as a dependency to Django.

[Chris Lamb]

This doesn't yet for modules that are lazily loaded - need to hook into the request_finished signal to update the files inotify is waiting on, otherwise we will never update the watch list (as we are blocking on it forever). The inotify interface will get in the way a bit here - will probably require calling oll ourselves with an additional anonymous file descriptor which (if written to) will tell us to update our watch list.

[Chris Lamb]

Update 0002-Add-support-for-pyinotify.patch

[Chris Lamb]

No need for file descriptor nonsense; we can simply update the inotify watch list from the thread the request_finished callback is called from. Anyway, current patch fixes the lazily loaded modules issue.

[ch0wn]

Update for django 1.2.1 and pyinotify 0.7

[ch0wn]

I combined and updated the previous patches and attached one that works with django 1.2.1 and latest pyinotify.

[Unai Zalakain]

Updated pull request: ​https://github.com/django/django/pull/1731

[Aymeric Augustin]

I left a few comments on the pull request. Overall this looks pretty good.

It still needs documentation. I'm not sure how we can write tests.

[Unai Zalakain]

I squashed the suggested changes into the old commit, please, let me know your opinion.

I'm sure how to test this either. Were do you suggest to add documentation?

[Marc Tamlyn]

I would suggest docs in the runserver docs? ​https://docs.djangoproject.com/en/dev/ref/django-admin/#runserver-port-or-address-port

[Unai Zalakain]

OK, added a short sentence stating that pyinotify is used for autoreloading when available.

[Tim Graham]

Please don't mark your own patch ​Ready for Checkin -- another person who reviews the patch should do that. I think this could use a bit more documentation as described in the PR.

[Unai Zalakain]

Sorry for the bad ticket handling, won't happen again. PR updated with suggested changes.

[Aymeric Augustin]

The pull request looks good to me.

However I can't really try it because I'm not using Linux for my development environment.

I would also suggest that the committer summarize and tone down the justification for the feature; it's a bit over the top right now eg. "Django is killing kittens". I get the humor, but not everyone will!

[Aymeric Augustin]

(I'm interested because I'd like to implement the equivalent behavior with kqueue on OS X, and this patch lays the groundwork.)

[Unai Zalakain]

When I copied the original justification, I wasn't sure what to do with that sentence :p It's now deleted ;)

[Claude Paroz]

Apart from minor cleanup with trailing spaces, I can confirm the patch is working well on my Linux machine.

[Tim Graham <timograham@…>]

In 15f82c701161b327cef54b469c00b6cbe01534db:

Fixed #9722 - used pyinotify as change detection system when available

Used pyinotify (when available) to replace the "pool-every-one-second"
mechanism in django.utils.autoreload.

Thanks Chris Lamb and Pascal Hartig for work on the patch.

[Tim Graham]

As noted in #22958, this doesn't seem to actually be improving performance. I'll revert it before the 1.7 release if no one wants to try to fix it before then.

[Claude Paroz]

Tentative pull request (tested that performance regression is gone for my use case):
​https://github.com/django/django/pull/2883

[Unai Zalakain]

Thank you for your time claudep!

[Tim Graham]

Looks good.

[Claude Paroz <claude@…>]

In 6d302f6396b7531eaa22dc5b544d233334e6fc92:

Fixed pyinotify performance regression in 15f82c7011

Refs #9722. Thanks Tim Graham for the review.

[Claude Paroz <claude@…>]

In 1bb8ccdb9e70bee35759f2ada4d661481852ab95:

[1.7.x] Fixed pyinotify performance regression in 15f82c7011

Refs #9722. Thanks Tim Graham for the review.
Backport of 6d302f639 from master.

[Tim Graham <timograham@…>]

In c8720e76:

Fixed #27685 -- Added watchman support to the autoreloader.

Removed support for pyinotify (refs #9722).