#16671 closed New feature (fixed)
5th tutorial on turning Polls into a reusable app
Reported by: | stumbles | Owned by: | |
---|---|---|---|
Component: | Documentation | Version: | dev |
Severity: | Normal | Keywords: | |
Cc: | katie@…, taavi@…, kmike84@…, ognajd@…, timograham@…, Matthias Dahl | Triage Stage: | Accepted |
Has patch: | yes | Needs documentation: | no |
Needs tests: | no | Patch needs improvement: | no |
Easy pickings: | no | UI/UX: | no |
Description
Katie Miller and I have been sprinting to create a tutorial to cover:
- reusable apps
- turning Polls into a reusable app
- Python packaging
- publishing your package
First draft attached.
Attachments (18)
Change History (55)
comment:1 by , 13 years ago
Patch needs improvement: | set |
---|---|
Triage Stage: | Unreviewed → Accepted |
Type: | Uncategorized → New feature |
comment:2 by , 13 years ago
Cc: | added |
---|
comment:3 by , 13 years ago
I've now added an updated patch to include the changes recommended by aaugustin. Sorry for the long delay.
comment:4 by , 13 years ago
Patch needs improvement: | unset |
---|
by , 13 years ago
Attachment: | tutorial05.2.diff added |
---|
by , 13 years ago
Attachment: | tutorial05.3.diff added |
---|
Extended README.txt, include "docs" in package and update tutorial index links.
by , 13 years ago
Attachment: | tutorial05.7.diff added |
---|
Closing bracked added previously was not quite in the right place.
comment:5 by , 13 years ago
Version: | 1.3 → SVN |
---|
I've now updated the project layout of this tutorial to match the changes in Django 1.4. I also repeated the previous tutorials again myself which helped clarify a few issues, such as where the project-wide templates directory will be after completing tutorials 1-4.
comment:6 by , 13 years ago
Cc: | added |
---|
Good work! Some notes:
- Templates won't be included in distribution (they should be added to package_data in setup.py);
- but I think it is bad idea to include templates in polls app because they usually tend to be too site-specific and overriding bundled templates is not always obvious;
- a couple of simple tests are must for a reusable app.
by , 13 years ago
Attachment: | tutorial05.10.diff added |
---|
Update tutorial to include templates directory via MANIFEST.in.
comment:7 by , 13 years ago
I've added necessary instructions to include templates directory in the package. For illustrative purposes, I think it is useful to show the reader how to include the templates.
The alternative would be to:
- remove the templates directory
- remove the "Completing your reusable app" section of the tutorial
- explain to the reader that people may find the templates too site-specific and difficult to override and that it may be better not to include them at all
- get the reader to add instructions to the package on how to write templates
Plus it's no longer self-contained, so not quite as satisfying.
Sure it may not be best practise to include templates, but to teach the basic concepts, I think this is an acceptable compromise.
comment:8 by , 13 years ago
Regarding tests, a doc test or unit test on models.Poll.was_published_today
might be a useful example. What other tests do you think should be added?
comment:9 by , 12 years ago
Just spoken to Russell at PyCon Australia sprints. He suggests that testing should be a separate tutorial of its own, rather than extending this one.
comment:10 by , 12 years ago
Hey Ben & Katie,
This has sat for a long while and I'd like to try to get this committed! Couple of comments:
- When I move the templates polls directory into the app folder, I had to add '/dir/to/mysite/' to settings.TEMPLATE_DIRS, previously I had only /path/to/mysite/mytemplates/ there.
- The django-polls/docs isn't included in the package even though it's included in MANIFEST.in unless it has a file in it. Perhaps a note could be added about that.
- Rather than suggest possible directories for dist/site-packages, use the command noted here: https://docs.djangoproject.com/en/dev/topics/install/#remove-any-old-versions-of-django
Nice work!
Tim
comment:11 by , 12 years ago
I'll add that for dist-packages the command in the link I noted above output "/usr/lib/python2.7/dist-packages" while polls was actually installed in "/usr/local/lib/python2.7". I discovered the actual location by running this command:
>>> import polls >>> polls <module 'polls' from '/usr/local/lib/python2.7/dist-packages/polls/__init__.pyc'>
So perhaps that should be recommended as an alternative?
by , 12 years ago
Attachment: | tutorial05.12.diff added |
---|
Note about empty docs dir and easy way to find location of installed package
comment:12 by , 12 years ago
Thanks for your notes Tim. Also looking forward to getting this in!
Regarding 1: I've just run through the dev version tutorials 1-4 and didn't find I needed to modify settings.TEMPLATE_DIRS
. If you have 'django.template.loaders.app_directories.Loader',
in your settings.TEMPLATE_LOADERS
(the default), the templates should be found by Django when moved inside polls/templates/polls
. Have I missed something?
Regarding 2: I've added a note about the docs
directory not being included if empty, thanks.
Regarding 3: I've used the example you mentioned in comment 11 to describe how to find package files to delete.
comment:13 by , 12 years ago
Also, I just noticed that while the templates directory is included in the package, it's not installed when I run python setup.py install
.
After some hunting, I learnt that one way to ensure the templates are installed is to use packages=setuptools.find_packages()
and include_package_data=True
in my setup.py file as follows:
from distutils.core import setup from setuptools import find_packages setup( name='django-polls', version='0.1', packages=find_packages(), license='', long_description=open('README.txt').read(), url='http://www.example.com/', author='Your Name', author_email='yourname@example.com', include_package_data=True, )
Is this the approach you would recommend? I'm not very experienced with packaging and noticed that the install behaved differently using after making these changes -- the installed files ended up in django_polls-0.1-py2.7.egg
and easy-install.pth
rather than the previous polls
and django_polls-0.1.egg-info
.
comment:14 by , 12 years ago
Replying to answer my own question - the following looks like a neater approach:
from distutils.core import setup setup( name='django-polls', version='0.1', packages=['polls'], package_data={'polls': ['templates/polls/*']}, license='', long_description=open('README.txt').read(), url='http://www.example.com/', author='Your Name', author_email='yourname@example.com', )
Please let me know if there's a better approach.
by , 12 years ago
Attachment: | tutorial05.13.diff added |
---|
Specify "package_data" explicitly and remove "include_package_data" from setup.py.
comment:15 by , 12 years ago
Looks good. Trying again, yes, it looks like the TEMPLATE_DIRS change isn't needed.
Made a couple tweaks:
"In the Tutorial 3" -> "In Tutorial 3"
Under the virtualenv section, it says "This has some" and looks like "disadvantages" got lost (added back).
I've built this and stuck it up a server temporarily for review. I'm going to email django-developers so we can get some final feedback.
by , 12 years ago
Attachment: | tutorial05.14.diff added |
---|
comment:16 by , 12 years ago
Very minor additional changes:
- use url function in README rather than tuple - seems this is now the default approach for new projects
- change "cd django-polls" to "cd django-polls-0.1"
by , 12 years ago
Attachment: | tutorial05.15.diff added |
---|
Use url fn in README and add version number when cd'ing
comment:17 by , 12 years ago
Just read through the tutorial. It looks good, but I don't know if it's the right thing to be teaching to someone at this point in their Django career. Reusable apps are something I would consider an advanced topic to be included as the final tutorial entry, if the topic belongs there at all. If you look at who the tutorial is geared at (someone definitely new to Django, but probably also new to Python), this tutorial is going to leave them scratching their head. What does in to the setup.py
? (Can I include my own URL, or does it have to be example.com? Does every reusable app for Django have to have django-
in front of the directory name?) What does the MANIFEST.in
file do and why is it there? (Do .in
files have some special meaning in Python in general? Can they do other things?)
In it's current state, I'm -1. I think this is a good foundation, but it needs to be fleshed out quiet a bit before it should be merged. I'm also -0 on whether reusable apps should be part 5 of the tutorial. I think tutorial 4's current list of next topics provides a really good starting point for further tutorials and that this is best left to a final spot or possibly even as part of the main documentation rather than the tutorial.
comment:18 by , 12 years ago
Cc: | added |
---|
I am +1 on this. One of the biggest challenges I faced when starting with Django is figuring out how to structure my projects/app and this tutorial clearly helps with that. Thank you to the authors working on this.
I suggest it ought to use distribute
which is a replacement for setup_tools
which was a successor for distuitls
- python package installation landscape is hairy to say the least: Differences between distribute, distutils and setuptools?. I suggest the following setup.py
.
from setuptools import setup, find_packages setup( install_requires=['distribute'], # let's require the successor to setuptools name='django-polls', version='0.1', packages=find_packages(), include_package_data=True, # this will use MANIFEST.in during install where we specify additional files license='', description=package.__doc__.strip(), long_description=open('README.txt').read(), url='http://www.example.com/', author='Your Name', author_email='yourname@example.com', )
I am torn between using very explicit package_data
or the include_package_data
and MANIFEST.in
to specify additional directories files. Former is more python however personally and from experience I settled on the latter as that is the only way to include root package files like README.txt
and is the simplest - Including Data Files.
include LICENSE include README.rst recursive-include polls/templates * recursive-include docs *
Also let's link to http://packages.python.org/distribute/index.html in the tutorial
comment:19 by , 12 years ago
I can see the problem of putting this into Tutorial 5. This stuff is really advanced if all you wanted was creating a simple web app. For that purpose "Tutorial 5: How to use reusable apps others wrote" would be much better.
So, could this be "Advanced tutorial: How to write reusable apps" or something like that.
One sign of problem is that I can't actually review the content of this tutorial for correctness, as I don't know the stuff this tutorial is introducing (I probably should, but that is another matter). Is this really something a newbie needs to be exposed on day 1?
comment:20 by , 12 years ago
Patch needs improvement: | set |
---|
I think positioning this as an "advanced tutorial" and incorporating some of the feedback above, as well as from the thread on django-developers* makes sense.
comment:21 by , 12 years ago
Cc: | added |
---|---|
Patch needs improvement: | unset |
Thanks for the feedback guys. I've updated the patch in the following ways:
- Included this as a separate "Advanced tutorial" rather than simply "Tutorial 5"
- Added text and/or links to answer questions like:
- What's the purpose of setup.py?
- What does the MANIFEST.in file do and why is it there?
- Does every reusable app for Django have to have django- in front of the directory name?
- Added a note that discusses naming conventions of packages.
- Added links to a couple more resources like readthedocs and djangopackages.com
You can view the HTML here: http://techytim.com/django/tutorial05/intro/reusable_apps.html
Since Django itself uses distutils and we have a tested patch using that, I think it makes to stick with that over distribute, although I am open to +1/-1 from any core devs.
The patch is based on the patch for #18715 Tutorial 3 refactor. Would appreciate a second set of eyes on that as well if you are interested.
comment:22 by , 12 years ago
Another good looking documentation addition. Some notes:
- The shoutout to djangopackages.com -- I agree this is the best resource in this particular space at the moment -- but that's *at the moment*. There have been others in the past, and as a project, we dont' have any guarantees that djangopackages will be around next week. This may be a case of something that we need to fix at a project level by formalizing the relationship with djangopackages, rather than changing the text here.
- In crossover from my comments on #18715; The introduction of app-level template directories seems more appropriate here, but it might still be worth a shoutout in Tutorial 3 letting people know that this option exists (even if you don't go into details)
- The "Why nested" note -- an example would help here. Show how a template name like "base.html" would easily collide between apps.
- Regarding distutils vs distribute: This is one of those areas where I think Django has a responsibility to be a community leader. The Hitchhikers guide to Python Packaging (which is itself worth a callout in the docs) recommends distribute; and everything I've seen says that the features of distribute will ultimately become part of distutils2; it seems to me that recommending distribute is the right thing to do here. AFAIK, the only reason Django itself doesn't use distribute is historical; setuptools had some pathological behavior back in the day, and nobody has had enough of an itch to make the change. Call this a +0 for distribute from me.
- The packaging says to create a README; it doesn't say create a LICENSE file. We *must* say this. Code without a license is *useless*. Simliarly, the setup.py file doesn't specify a license definition. By way of communtiy guidance, we can say "Django, and many Django-compatible apps are distributed under the BSD license; however, you're free to pick your own license. However, be aware that your licensing choice will affect who is able to use your code".
- The setup.py also doesn't mention any trove classifiers. It should. At the very least, we should be able to say:
'Environment :: Web Environment', 'Framework :: Django', 'Intended Audience :: Developers', 'License :: OSI Approved :: BSD License', 'Operating System :: OS Independent', 'Programming Language :: Python', 'Programming Language :: Python :: 2.6', 'Programming Language :: Python :: 2.7', 'Topic :: Internet :: WWW/HTTP', 'Topic :: Internet :: WWW/HTTP :: Dynamic Content',
- Again, I'm a little hesitant to link out to third-party blogs and projects in the "more about" section. If we need to elaborate best practices, I'd say we should distil Eric's blog post into a couple of paragraphs of best practices; if we need to link to a list of packages, djangopackages.com would seem a better candidate (with the caveat from above).
comment:23 by , 12 years ago
Agreed, this is looking great.
I don't have clarity on how best to both keep it simple and recommend best practices in the morass of confusion that is Python packaging, when it comes to recommending distutils vs distribute. You can't really recommend distribute without explaining that it's a third-party package, not part of the Python standard library, meaning that you're preventing people who don't have it from installing your package - but at the same time, in practice everybody has it, so that's not really a problem. But that's an awful lot of extra potentially-very-confusing information to try to squeeze in (not to mention explaining _why_ you'd use distribute rather than the stdlib option), all of which argues for just using distutils.
The main problem I see is in recommending distutils and recommending "python setup.py install" in the same tutorial - because that's the one path that means no useful metadata is installed along with the package, and therefore automated uninstall is not possible (pip is not able to uninstall packages installed in this way). When you add in "sudo", it means our tutorial is telling people to install a test package as the root user, to their system site-packages, in such a way that it can't be removed without manually sudoing and using "rm -rf". That's... unfortunate. We have three choices (use distutils, demonstrate "python setup.py install", default to "sudo" rather than virtualenv) which I think are all individually the right choice for simplicity's sake, but the combination has a sub-optimal result. If we had to change any of those three, I'd probably go for switching from distutils to distribute.
Regardless, if the tutorial tells them to "sudo python setup.py install", I think we need to tell them how they can uninstall it. If we use distribute, we can just link to pip and say "pip can uninstall this." If we stick with distutils, we have to tell them they'd need to manually "sudo rm -rf" the polls directory (which they can find as demonstrated, by importing polls and checking polls.__file__
).
One other note - in Python 2.7 it is no longer necessary to redundantly add files to MANIFEST.in that are listed in package_data
(in this case, the templates). That was a bug in distutils in Python 2.6 and previous that was fixed in Python 2.7. I guess as long as we support 2.6 we should probably demonstrate a 2.6-compatible technique, but I wonder if it's worth a brief aside to note the redundancy, and that you don't need it if generating the sdist using Python 2.7. In any case, whenever Django drops 2.6 support this should be changed.
And lastly: if we keep the link to Eric's article, his last name is spelled "Holscher".
comment:24 by , 12 years ago
One other piece of feedback - I'm in complete agreement that this is really "advanced tutorial 1" rather than "tutorial 5"; so when committed, the tutorial URL should probably reflect that.
comment:25 by , 12 years ago
Patch needs improvement: | set |
---|
Thanks for the feedback, I'll work to incorporate these suggestions.
by , 12 years ago
Attachment: | tutorial05.17.diff added |
---|
comment:26 by , 12 years ago
Updates:
- Added a note in tutorial 3 regarding app-level template directories.
- Added an example of how template names could collide to the "Why nested" note.
- Changed from distutils to distribute and pip.
- Added a step of creating a license file
- Added trove classifiers to setup.py
- Removed link to Eric's blog post - if someone wants to add documentation on best practices it can be a separate ticket.
Other Notes:
- Since we switched to using distribute, it looks like Carl's point about redundancy between MANIFEST.in and package_data is moot.
- The updated doc is http://techytim.com/django/tutorial05/intro/reusable-apps.html
Todo:
- Should we keep the link to djangopackages.com?
- Russ, regarding the URL, I'm assuming you didn't like the 'intro' bit? Since this follows from tutorial 4, I felt like moving it to a whole new top level like /advanced-tutorials/reuseable-apps/ might be a bit too much (we won't get any next/previous links to and from tutorial 4) so I opted for /intro/reusable-apps/. I'm open to suggestions if you still don't like it. (And I just realized that maybe you saw tutorial05 in the url where I posted the built docs which is just a folder on my server where I put this.)
comment:27 by , 12 years ago
I'm happy keeping the link to djangopackages.com; it's a reasonably robust part of the community, and we can start negotiations with Danny about making the arrangement a little more formal.
And yes, my objection was to the tutorial05 part in the URL. If that's just a build artefact, that's fine - I just raising it in case it was the proposed location in the source tree.
comment:28 by , 12 years ago
Cc: | added |
---|
comment:29 by , 12 years ago
Some comments on the updated draft:
- Per its author, setuptools is not defunct, and he has made fixes in it since the distribute fork, so I think it's needlessly partisan to refer to it as "defunct" and claim that it has "bugs which will never be fixed." I recommend this alternative description of distribute: "It's a community-maintained fork of the older setuptools project." (Also, there's a mismatched parenthesis at the end of that sentence currently).
- I'm -0.5 on linking to the "state of Python packaging" diagram from the Hitchhiker's Guide, as it has some factual errors (pip doesn't work directly with distutils, it relies on setuptools/distribute) and makes assertions which at this point remain far from clear (i.e. that setuptools/distribute are headed for the dustbin, even though today they remain by far the most-used Python packaging tools, or even that distutils2 is headed for the stdlib - it was removed from Python 3.3 and so far there's been no movement to reintroduce it for Python 3.4, rather pieces of it are being rewritten under yet another name, "distlib"). On the other hand, I appreciate the attractiveness of having something simple to link to that demonstrates just how muddled things are, and I don't have a replacement link target to recommend in that role.
- Including distribute (or setuptools) in the
install_requires
insetup.py
is wrong (unless your package actually imports and uses "setuptools" or "pkg_resources" at runtime - not in setup.py but in the actual packaged code - which the "polls" app does not). The note below claims that "the install_requires line ensure that distribute will be installed for anyone who is installing our package" but this is not true - if someone doesn't have setuptools or distribute installed to begin with, setup.py won't even run long enough to get to theinstall_requires
, it'll raiseImportError
on the first line. And havingdistribute
ininstall_requires
means that if a user has setuptools installed, installing your package will forcibly replace it with distribute, which is really unfriendly - the point of distribute being a drop-in replacement is that your package should work equally well for people using setuptools or distribute, and should not enforce one choice or the other on your users.
- Using
open('README.txt').read()
as the value oflong_description
will break ifsetup.py
is executed from outside the current directory (e.g.python django-polls/setup.py install
). For this to be reliable, animport os
is needed up top, then ahere = os.path.dirname(os.path.abspath(__file__))
, thenlong_description=open(os.path.join(here, 'README.txt')).read()
.
- The line "only a limited set of files are included in the package by default" could be made more specific: "only Python modules and packages are included in the package by default."
Otherwise, looks good to me!
follow-up: 31 comment:30 by , 12 years ago
In trying to address the fourth bullet point, I updated long_description as you suggested but the package still doesn't install properly when running setup.py outside django-polls-0.1:
$ sudo python django-polls-0.1/setup.py install running install Checking .pth file support in /usr/local/lib/python2.7/dist-packages/ /usr/bin/python -E -c pass TEST PASSED: /usr/local/lib/python2.7/dist-packages/ appears to support .pth files running bdist_egg running egg_info writing django_polls.egg-info/PKG-INFO writing top-level names to django_polls.egg-info/top_level.txt writing dependency_links to django_polls.egg-info/dependency_links.txt warning: manifest_maker: standard file 'setup.py' not found reading manifest file 'django_polls.egg-info/SOURCES.txt' writing manifest file 'django_polls.egg-info/SOURCES.txt' installing library code to build/bdist.linux-i686/egg running install_lib warning: install_lib: 'build/lib.linux-i686-2.7' does not exist -- no Python modules to install creating build/bdist.linux-i686/egg creating build/bdist.linux-i686/egg/EGG-INFO copying django_polls.egg-info/PKG-INFO -> build/bdist.linux-i686/egg/EGG-INFO copying django_polls.egg-info/SOURCES.txt -> build/bdist.linux-i686/egg/EGG-INFO copying django_polls.egg-info/dependency_links.txt -> build/bdist.linux-i686/egg/EGG-INFO copying django_polls.egg-info/top_level.txt -> build/bdist.linux-i686/egg/EGG-INFO zip_safe flag not set; analyzing archive contents... creating 'dist/django_polls-0.1-py2.7.egg' and adding 'build/bdist.linux-i686/egg' to it removing 'build/bdist.linux-i686/egg' (and everything under it) Processing django_polls-0.1-py2.7.egg creating /usr/local/lib/python2.7/dist-packages/django_polls-0.1-py2.7.egg Extracting django_polls-0.1-py2.7.egg to /usr/local/lib/python2.7/dist-packages Adding django-polls 0.1 to easy-install.pth file Installed /usr/local/lib/python2.7/dist-packages/django_polls-0.1-py2.7.egg Processing dependencies for django-polls==0.1 Finished processing dependencies for django-polls==0.1
A directory is created: /usr/local/lib/python2.7/dist-packages/django_polls-0.1-py2.7.egg
but the polls subdirectory is missing.
comment:31 by , 12 years ago
Replying to timo:
In trying to address the fourth bullet point, I updated long_description as you suggested but the package still doesn't install properly when running setup.py outside django-polls-0.1:
...
A directory is created: /usr/local/lib/python2.7/dist-packages/django_polls-0.1-py2.7.egg
but the polls subdirectory is missing.
Interesting, I've never seen that. My first guess would be that it's a bug in find_packages
? I generally don't use that, even when using setuptools/distribute, I just list packages explicitly. Does removing find_packages
fix it?
comment:32 by , 12 years ago
With packages=['polls']
:
sudo python django-polls-0.1/setup.py install running install Checking .pth file support in /usr/local/lib/python2.7/dist-packages/ /usr/bin/python -E -c pass TEST PASSED: /usr/local/lib/python2.7/dist-packages/ appears to support .pth files running bdist_egg running egg_info creating django_polls.egg-info writing django_polls.egg-info/PKG-INFO writing top-level names to django_polls.egg-info/top_level.txt writing dependency_links to django_polls.egg-info/dependency_links.txt writing manifest file 'django_polls.egg-info/SOURCES.txt' warning: manifest_maker: standard file 'setup.py' not found error: package directory 'polls' does not exist
It may be a path issue. For example, if I rename django-polls-0.1
to django-polls
and then modify setup.py to packages=['django-polls/polls']
it works; of course, this breaks it when running setup.py in the django-polls directory.
Yes, if I add:
my_path = os.path.abspath(__file__) os.chdir(os.path.normpath(os.path.join(my_path, os.pardir)))
after here=os.path.dirname...
in setup.py, I can run setup.py from any path. Fun, fun.
comment:33 by , 12 years ago
Looks like even distutils itself is built to assume that setup.py is in the current working directory. Interesting, I didn't realize that. I think I'd never tried to actually run sdist or install from a different directory, only simple metadata-getting commands (python setup.py egg_info
or e.g. python setup.py --long-description
).
That definitely reduces the importance of point 4, but I still think it's worth doing this the robust way, because at least then a few more (simple) uses of setup.py will work from a different directory. I've definitely seen real bugs filed against real packages because the handling of README.txt
wasn't robust against execution from a different directory.
by , 12 years ago
Attachment: | tutorial05.18.diff added |
---|
comment:34 by , 12 years ago
Updates:
- Updated language for setuptools vs. distribute per Carl's suggestion
- Removed link to "state of python packaging" due to inaccuracies.
- Removed
install_requires=['distribute']
from setup.py - Updated
open('README.txt')
to allow executing setup.py from any directory - Made language regarding what files are included by default more specific per Carl's suggestion
comment:35 by , 12 years ago
Patch needs improvement: | unset |
---|
comment:36 by , 12 years ago
Resolution: | → fixed |
---|---|
Status: | new → closed |
This is a good addition to the docs.
I skimmed through the patch and I suggest the following improvements: