Document the possibility of using egg-files for apps / reintroduce support for setuptools to continue the reusable apps GSoC project

[Jannis Leidel]

Setuptools' pkg_resources allows module loading from egg-files, which is also applicable for Django apps. The attached code enables Django to load apps from locations defined in a APP_DIRS list. All modules having the entry_point defined in APP_ENTRY_POINT (default: django.apps) found in these directories are automatically loaded and added to the INSTALLED_APPS list.

Please keep in mind to use the 'django.template.loaders.eggs.load_template_source' in TEMPLATE_LOADERS to load templates from eggs.

More information on using Django with setuptools in my weblog entry: ​http://jannisleidel.com/2007/11/using-django-with-setuptools/

[Jannis Leidel]

first implementation

[Jacob]

Couple of comments:

• This needs tests, and since I roll egg-free I don't really have the knowledge to write it myself.
• I don't like the two new settings. Is there any reason not to simply let entries in INSTALLED_APPS refer to packages that are eggs on PYTHONPATH?

Other than that, this looks good.

[Jannis Leidel]

1. Indeed, tests will follow.
2. Nothing speaks against just loading egg-based apps from the python path. The idea of the entry point is to automate the addition of the app path to PYTHONPATH - so it's merely a helper, not a requirement. The docs should reflect that, you are right.

My idea of egg-loading (even part of the code) comes from Trac's implementation of "components" which just need to be dropped in a component directory. The APP_LIST setting provides a way to for pkg_resources to look for the entrypoints. This is of course very useful for a Django apps repository later.

[Jacob]

As far as I'm concerned, there's no reason to load eggs that aren't on PYTHONPATH. Why reinvent the wheel -- PYTHONPATH is the blsessed method for locating Python code, and there's no reason for us to do it any differently.

[Jannis Leidel]

First and foremost, I really meant "APP_DIRS", not "APP_LIST" in my previous comment, I certainly was too tired that moment :)

Agreed, the PYTHONPATH is for me also *the way* to handle python module, cough, Django apps. I hoped to make things "easier" for beginners by providing the option to download a ready-made application-egg and just drop it in one of the APP_DIRS. Thanks to the entry_point mechanism the rest would be automatic. But this is too much magic, I admit.

For the sake of simplicity I'll change the patch to just include documentation about the possibility to put any egg-based module in INSTALLED_APPS. The changed setup.py would be great to have, though.

[Jannis Leidel]

Removed "pkg_resources app loading with entrypoints"-code from first patch and added more documentation, since it's enough to have an egg file on the PYTHONPATH. kudos to jacob for bringing me back down to earth :)

[Jannis Leidel]

Only use setuptools if installed, otherwise fallback to distuils (now without throwing a UserWarning because of the unsupported setup keywords), fully backwards compatible from my point of view

[Jannis Leidel]

Please note that the last patch (reusableapps3.diff) contains documentation which is only valid if:

• the Django entry in the PyPI is updated to include the download_url variable
• the Django download section (​http://www.djangoproject.com/download/) includes:
  • links to the source packages with *.tar.gz or *.zip extensions, rather than providing absolute URLs like ​http://www.djangoproject.com/download/0.95.2/tarball/
  • a link to the Django subversion repository in the form: ​http://code.djangoproject.com/svn/django/trunk/#egg=Django-pre

[Jannis Leidel]

Regarding writing tests: Since there is no actual code change to Django in the last patch, I don't see a real reason for tests anymore.

[Jacob]

This patch to setup.py looks good and all, but that's not what this ticket is about -- did you maybe attach the wrong patch to the wrong ticket? If not, please separate adding setuptools into setup.py from the egg-app-loading mechanism this ticket was opened for.

And you do need tests, which I think are as simple as placing an egg in tests/regressiontests and making sure they it gets loaded correctly.

[Jannis Leidel]

Oops, I got the impression from your earlier comments that the loading mechanism from the first patch (looking for eggs with a specific entry point in a liste of directory paths) shouldn't be in Django because it's enough to find them anywhere on the PYTHONPATH. When setuptools are installed "it just works" and loads the eggs. Then INSTALLED_APPS can be used to load apps from eggs.

I understand that the changes to setup.py need to move to a new ticket. But let's figure out first, how we solve the misunderstandings from above.

Regarding the tests: good idea. I admit I'm not very used to write tests. But I'll do my best.

[Ben Slavin]

Unless I'm greatly misunderstanding something here, Django already has support for loading Django apps from egg-files because Python does it automatically... you don't even need to have setuptools installed for it to work (assuming the egg doesn't use any in-package data files).

Since no changes to Django are necessary to make this happen, the only change I see is a debatable documentation change (which happens to be in the wrong spot... should be in docs/settings.txt).

I'm even -0.5 on the documentation change since the current description ("full Python path to a Python package") applies to eggs as well.

The setuptools change should be opened as a separate ticket since it's not related to "loading Django apps from egg-files". Without having really examined that code, it looks like something worth doing.

[Jannis Leidel]

Unless I'm greatly misunderstanding something here, Django already has support for loading Django apps from egg-files because Python does it automatically... you don't even need to have setuptools installed for it to work (assuming the egg doesn't use any in-package data files)

You are right: Django supports loading Python Eggs just like any other Python-based application. But you hit the nail on the head with in-package data files (like templates): we should be able to put those in Django apps eggs.

Since no changes to Django are necessary to make this happen, the only change I see is a debatable documentation change (which happens to be in the wrong spot... should be in docs/settings.txt).

I'm even -0.5 on the documentation change since the current description ("full Python path to a Python package") applies to eggs as well.

I proposed (not without thinking about it a lot before - see my GSoC documentation) that future reusable Django apps (based on egg-files and a central web repository) should include a setuptools flag called "entry point". That would easily allow Django to load an app from directories not on the PYTHONPATH. As I stated earlier, I think removing this code was a misunderstanding with Jacob during the Sprint and needs clarification. I know that this seems to be reinventing the wheel, on first sight.

Since Django beginners are often new to Python as well, it should be in Django's documentation, even additionally in docs/settings.txt under INSTALLED_APPS.

The setuptools change should be opened as a separate ticket since it's not related to "loading Django apps from egg-files". Without having really examined that code, it looks like something worth doing.

I know, you'll get a new ticket once we discussed the real issue here. Probably I just don't want this topic to go down in a flame war about setuptools.

[Ben Slavin]

Replying to jezdez:

You are right: Django supports loading Python Eggs just like any other Python-based application. But you hit the nail on the head with in-package data files (like templates): we should be able to put those in Django apps eggs.

Django already contains the ability to load templates from eggs (django.template.loaders.eggs.load_template_source).

Does this current functionality not serve your needs? Is there some other use case that I'm overlooking?

[Jannis Leidel]

Django already contains the ability to load templates from eggs (django.template.loaders.eggs.load_template_source).

Templates are just one possible example.

Does this current functionality not serve your needs? Is there some other use case that I'm overlooking?

I think it's worthwhile to be able to include (site_)media in general, like images, stylesheets and javascript if we encourage people to build resuable apps (or will).

Though I bet there are lots of other kinds of package data which I can't think of bundling with Django apps right now. Fact is: we shouldn't prevent users to do so in the first place.

[Ben Slavin]

Replying to jezdez:

I think it's worthwhile to be able to include (site_)media in general, like images, stylesheets and javascript if we encourage people to build resuable apps (or will).

Templates are one thing, but static media is a different story all-together. Using Django to serve static media is explicitly discouraged... and would be required if such media is distributed in eggs.

However, these issues aren't related to this ticket.

Django can currently load apps from eggs and can even load templates from eggs, so we don't need to "add support for loading Django apps from egg-files".

If I'm reading the discussion here correctly, Jacob's blessing was issued because it seemed like loading from eggs didn't work. (Jacob, if I'm wrong please correct me.)

I'm very tempted to mark the ticket as worksforme based on this... but won't since Jacob marked it as 'accepted'. If you'd like to discuss bringing back setuptools or support for loading static media from eggs, please open tickets for those issues or bring it up for discussion on django-developers.

[Jannis Leidel]

Replying to __hawkeye__:

Replying to jezdez:

I think it's worthwhile to be able to include (site_)media in general, like images, stylesheets and javascript if we encourage people to build resuable apps (or will).

Templates are one thing, but static media is a different story all-together. Using Django to serve static media is explicitly discouraged... and would be required if such media is distributed in eggs.

Please reread my comment, there was no mention of serving static in-package files with Django.

However, these issues aren't related to this ticket.

Django can currently load apps from eggs and can even load templates from eggs, so we don't need to "add support for loading Django apps from egg-files".

As I said, one of the greater goals of this ticket is to tell people of the actual existence of eggs. I changed the topic to reflect that.

If I'm reading the discussion here correctly, Jacob's blessing was issued because it seemed like loading from eggs didn't work. (Jacob, if I'm wrong please correct me.)

I'm very tempted to mark the ticket as worksforme based on this... but won't since Jacob marked it as 'accepted'. If you'd like to discuss bringing back setuptools or support for loading static media from eggs, please open tickets for those issues or bring it up for discussion on django-developers.

You must have missed my mails. I brought this up on the mailing list more than once. During and after the summer of code, during the first and the second sprint (this ticket).

[Jannis Leidel]

I really don't want to spawn another discussion about setuptools' hackiness but wonder if we could have some documentation and the small changes to setup.py to make it easier for users to install Django with setuptools. It really has become the standard, so what's the plan?

[(none)]

Milestone post-1.0 deleted

[Jannis Leidel]

I hope everone understands, but I learned from my mistakes and seriously consider the way setuptools hacks distutils as evil now -- hence closing this as wontfix. Tools like pip/virtualenv and buildout solve the problems of software packages in daily work better than easy_install/setuptools alone -- by adding more meaning to the process of deployment and environmental encapsulation.