Opened 16 years ago

Closed 14 years ago

#6080 closed (wontfix)

Document the possibility of using egg-files for apps / reintroduce support for setuptools to continue the reusable apps GSoC project

Reported by: Jannis Leidel Owned by: Jannis Leidel
Component: Core (Other) Version: dev
Severity: Keywords: setuptools apps eggs sprintdec01
Cc: Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: yes Patch needs improvement: yes
Easy pickings: no UI/UX: no

Description

Setuptools' pkg_resources allows module loading from egg-files, which is also applicable for Django apps. The attached code enables Django to load apps from locations defined in a APP_DIRS list. All modules having the entry_point defined in APP_ENTRY_POINT (default: django.apps) found in these directories are automatically loaded and added to the INSTALLED_APPS list.

Please keep in mind to use the 'django.template.loaders.eggs.load_template_source' in TEMPLATE_LOADERS to load templates from eggs.

More information on using Django with setuptools in my weblog entry: http://jannisleidel.com/2007/11/using-django-with-setuptools/

Attachments (3)

reusableapps.diff (8.4 KB ) - added by Jannis Leidel 16 years ago.
first implementation
reusableapps2.diff (6.7 KB ) - added by Jannis Leidel 16 years ago.
Removed "pkg_resources app loading with entrypoints"-code from first patch and added more documentation, since it's enough to have an egg file on the PYTHONPATH. kudos to jacob for bringing me back down to earth :)
reusableapps3.diff (7.8 KB ) - added by Jannis Leidel 16 years ago.
Only use setuptools if installed, otherwise fallback to distuils (now without throwing a UserWarning because of the unsupported setup keywords), fully backwards compatible from my point of view

Download all attachments as: .zip

Change History (23)

by Jannis Leidel, 16 years ago

Attachment: reusableapps.diff added

first implementation

comment:1 by Jacob, 16 years ago

Needs tests: set
Patch needs improvement: set
Triage Stage: UnreviewedAccepted

Couple of comments:

  • This needs tests, and since I roll egg-free I don't really have the knowledge to write it myself.
  • I don't like the two new settings. Is there any reason not to simply let entries in INSTALLED_APPS refer to packages that are eggs on PYTHONPATH?

Other than that, this looks good.

comment:2 by Jannis Leidel, 16 years ago

  1. Indeed, tests will follow.
  2. Nothing speaks against just loading egg-based apps from the python path. The idea of the entry point is to automate the addition of the app path to PYTHONPATH - so it's merely a helper, not a requirement. The docs should reflect that, you are right.

My idea of egg-loading (even part of the code) comes from Trac's implementation of "components" which just need to be dropped in a component directory. The APP_LIST setting provides a way to for pkg_resources to look for the entrypoints. This is of course very useful for a Django apps repository later.

comment:3 by Jacob, 16 years ago

As far as I'm concerned, there's no reason to load eggs that aren't on PYTHONPATH. Why reinvent the wheel -- PYTHONPATH is the blsessed method for locating Python code, and there's no reason for us to do it any differently.

comment:4 by Jannis Leidel, 16 years ago

First and foremost, I really meant "APP_DIRS", not "APP_LIST" in my previous comment, I certainly was too tired that moment :)

Agreed, the PYTHONPATH is for me also *the way* to handle python module, cough, Django apps. I hoped to make things "easier" for beginners by providing the option to download a ready-made application-egg and just drop it in one of the APP_DIRS. Thanks to the entry_point mechanism the rest would be automatic. But this is too much magic, I admit.

For the sake of simplicity I'll change the patch to just include documentation about the possibility to put any egg-based module in INSTALLED_APPS. The changed setup.py would be great to have, though.

by Jannis Leidel, 16 years ago

Attachment: reusableapps2.diff added

Removed "pkg_resources app loading with entrypoints"-code from first patch and added more documentation, since it's enough to have an egg file on the PYTHONPATH. kudos to jacob for bringing me back down to earth :)

by Jannis Leidel, 16 years ago

Attachment: reusableapps3.diff added

Only use setuptools if installed, otherwise fallback to distuils (now without throwing a UserWarning because of the unsupported setup keywords), fully backwards compatible from my point of view

comment:5 by Jannis Leidel, 16 years ago

Please note that the last patch (reusableapps3.diff) contains documentation which is only valid if:

comment:6 by Jannis Leidel, 16 years ago

Regarding writing tests: Since there is no actual code change to Django in the last patch, I don't see a real reason for tests anymore.

comment:7 by Jacob, 16 years ago

This patch to setup.py looks good and all, but that's not what this ticket is about -- did you maybe attach the wrong patch to the wrong ticket? If not, please separate adding setuptools into setup.py from the egg-app-loading mechanism this ticket was opened for.

And you do need tests, which I think are as simple as placing an egg in tests/regressiontests and making sure they it gets loaded correctly.

comment:8 by Jannis Leidel, 16 years ago

Oops, I got the impression from your earlier comments that the loading mechanism from the first patch (looking for eggs with a specific entry point in a liste of directory paths) shouldn't be in Django because it's enough to find them anywhere on the PYTHONPATH. When setuptools are installed "it just works" and loads the eggs. Then INSTALLED_APPS can be used to load apps from eggs.

I understand that the changes to setup.py need to move to a new ticket. But let's figure out first, how we solve the misunderstandings from above.

Regarding the tests: good idea. I admit I'm not very used to write tests. But I'll do my best.

comment:9 by Ben Slavin, 16 years ago

Unless I'm greatly misunderstanding something here, Django already has support for loading Django apps from egg-files because Python does it automatically... you don't even need to have setuptools installed for it to work (assuming the egg doesn't use any in-package data files).

Since no changes to Django are necessary to make this happen, the only change I see is a debatable documentation change (which happens to be in the wrong spot... should be in docs/settings.txt).

I'm even -0.5 on the documentation change since the current description ("full Python path to a Python package") applies to eggs as well.

The setuptools change should be opened as a separate ticket since it's not related to "loading Django apps from egg-files". Without having really examined that code, it looks like something worth doing.

in reply to:  9 ; comment:10 by Jannis Leidel, 16 years ago

Unless I'm greatly misunderstanding something here, Django already has support for loading Django apps from egg-files because Python does it automatically... you don't even need to have setuptools installed for it to work (assuming the egg doesn't use any in-package data files)


You are right: Django supports loading Python Eggs just like any other Python-based application. But you hit the nail on the head with in-package data files (like templates): we should be able to put those in Django apps eggs.


Since no changes to Django are necessary to make this happen, the only change I see is a debatable documentation change (which happens to be in the wrong spot... should be in docs/settings.txt).

I'm even -0.5 on the documentation change since the current description ("full Python path to a Python package") applies to eggs as well.


I proposed (not without thinking about it a lot before - see my GSoC documentation) that future reusable Django apps (based on egg-files and a central web repository) should include a setuptools flag called "entry point". That would easily allow Django to load an app from directories not on the PYTHONPATH. As I stated earlier, I think removing this code was a misunderstanding with Jacob during the Sprint and needs clarification. I know that this seems to be reinventing the wheel, on first sight.

Since Django beginners are often new to Python as well, it should be in Django's documentation, even additionally in docs/settings.txt under INSTALLED_APPS.


The setuptools change should be opened as a separate ticket since it's not related to "loading Django apps from egg-files". Without having really examined that code, it looks like something worth doing.


I know, you'll get a new ticket once we discussed the real issue here. Probably I just don't want this topic to go down in a flame war about setuptools.

in reply to:  10 ; comment:11 by Ben Slavin, 16 years ago

Replying to jezdez:

You are right: Django supports loading Python Eggs just like any other Python-based application. But you hit the nail on the head with in-package data files (like templates): we should be able to put those in Django apps eggs.


Django already contains the ability to load templates from eggs (django.template.loaders.eggs.load_template_source).

Does this current functionality not serve your needs? Is there some other use case that I'm overlooking?

in reply to:  11 ; comment:12 by Jannis Leidel, 16 years ago

Django already contains the ability to load templates from eggs (django.template.loaders.eggs.load_template_source).

Templates are just one possible example.


Does this current functionality not serve your needs? Is there some other use case that I'm overlooking?

I think it's worthwhile to be able to include (site_)media in general, like images, stylesheets and javascript if we encourage people to build resuable apps (or will).

Though I bet there are lots of other kinds of package data which I can't think of bundling with Django apps right now. Fact is: we shouldn't prevent users to do so in the first place.

in reply to:  12 ; comment:13 by Ben Slavin, 16 years ago

Replying to jezdez:

I think it's worthwhile to be able to include (site_)media in general, like images, stylesheets and javascript if we encourage people to build resuable apps (or will).


Templates are one thing, but static media is a different story all-together. Using Django to serve static media is explicitly discouraged... and would be required if such media is distributed in eggs.

However, these issues aren't related to this ticket.

Django can currently load apps from eggs and can even load templates from eggs, so we don't need to "add support for loading Django apps from egg-files".

If I'm reading the discussion here correctly, Jacob's blessing was issued because it seemed like loading from eggs didn't work. (Jacob, if I'm wrong please correct me.)

I'm very tempted to mark the ticket as worksforme based on this... but won't since Jacob marked it as 'accepted'. If you'd like to discuss bringing back setuptools or support for loading static media from eggs, please open tickets for those issues or bring it up for discussion on django-developers.

in reply to:  13 comment:14 by Jannis Leidel, 16 years ago

Summary: Add support for loading Django apps from egg-filesAdd documentation about the possibility to use egg-files as Django apps in preparation of the hotclub of France and reintroduce support for setuptools in continuation of the reusable DJango apps summer of code project (was: Add support for loading Django apps from egg-files)

Replying to __hawkeye__:

Replying to jezdez:

I think it's worthwhile to be able to include (site_)media in general, like images, stylesheets and javascript if we encourage people to build resuable apps (or will).


Templates are one thing, but static media is a different story all-together. Using Django to serve static media is explicitly discouraged... and would be required if such media is distributed in eggs.

Please reread my comment, there was no mention of serving static in-package files with Django.

However, these issues aren't related to this ticket.

Django can currently load apps from eggs and can even load templates from eggs, so we don't need to "add support for loading Django apps from egg-files".

As I said, one of the greater goals of this ticket is to tell people of the actual existence of eggs. I changed the topic to reflect that.

If I'm reading the discussion here correctly, Jacob's blessing was issued because it seemed like loading from eggs didn't work. (Jacob, if I'm wrong please correct me.)

I'm very tempted to mark the ticket as worksforme based on this... but won't since Jacob marked it as 'accepted'. If you'd like to discuss bringing back setuptools or support for loading static media from eggs, please open tickets for those issues or bring it up for discussion on django-developers.

You must have missed my mails. I brought this up on the mailing list more than once. During and after the summer of code, during the first and the second sprint (this ticket).

comment:15 by Jannis Leidel, 16 years ago

Summary: Add documentation about the possibility to use egg-files as Django apps in preparation of the hotclub of France and reintroduce support for setuptools in continuation of the reusable DJango apps summer of code project (was: Add support for loading Django apps from egg-files)Add documentation about the possibility to use egg-files as Django apps in preparation of the hotclub of France and reintroduce support for setuptools in continuation of the reusable Django apps summer of code project (was: Add support for loading Django apps from egg-files)

comment:16 by anonymous, 16 years ago

Summary: Add documentation about the possibility to use egg-files as Django apps in preparation of the hotclub of France and reintroduce support for setuptools in continuation of the reusable Django apps summer of code project (was: Add support for loading Django apps from egg-files)Document the possibility of using egg-files for apps / reintroduce support for setuptools to continue the reusable apps GSoC project

comment:17 by Jannis Leidel, 16 years ago

milestone: 1.0

I really don't want to spawn another discussion about setuptools' hackiness but wonder if we could have some documentation and the small changes to setup.py to make it easier for users to install Django with setuptools. It really has become the standard, so what's the plan?

comment:18 by Malcolm Tredinnick, 16 years ago

milestone: 1.0post-1.0

comment:19 by (none), 15 years ago

milestone: post-1.0

Milestone post-1.0 deleted

comment:20 by Jannis Leidel, 14 years ago

Resolution: wontfix
Status: newclosed

I hope everone understands, but I learned from my mistakes and seriously consider the way setuptools hacks distutils as evil now -- hence closing this as wontfix. Tools like pip/virtualenv and buildout solve the problems of software packages in daily work better than easy_install/setuptools alone -- by adding more meaning to the process of deployment and environmental encapsulation.

Note: See TracTickets for help on using tickets.
Back to Top