New releases do not reliably overwrite old ones

[holdenweb]

There have been several issues reported where .pyc files from older releases apparently outlive the installation of a newer release.

The reason for this is that currently releases appear to be cut without altering the modification date of distributed files. Thus a .py file has not been changed since Jan 12, 2007 will appear in the release with that modification date.

When the release is installed with distutils the new .py files will replace their older versions, but if the .pyc from the prior release was compiled *after* the new .py file was last modified in SVN the new file will appear older than its corresponding .pyc, and the compiled file from the prior release will be retained, to the consternation of the installing user.

The simplest fix for this would appear to be to change the release engineering process so that after extraction from SVN all source files are touched to bring their modification date close to that of the release.

Strictly speaking this might also be considered a distutils bug, but the issue has not been reported elsewhere.

[Karen Tracey]

#9090 is one symptom of this problem. As described there, the problem is not that the .pyc files are not updated but that the newer .py files are not copied to the installation directory. So even clearing out all the .pyc files in the install tree after this has happened won't produce a consistent level, since you've got a mixture of old and new .py files installed. Also, more discussion is here: ​http://groups.google.com/group/django-developers/browse_thread/thread/85d9500f319369a1. The problem seems to be resulting from old release tarfiles using tarfile creation date for contained files while 1.0 has last-modified-in-svn dates. That makes some files in the 1.0 release appear to be older than what was previously installed. (I do not think there is a distutils bug here.)

[(none)]

Milestone post-1.0 deleted

[Jacob]

So what's the fix here? Touch all the files as part of rolling the distribution (i.e. from setup.py sdist)?

[Aymeric Augustin]

Given Karen's last message in the thread linked above, the safest solution is to ensure all files in the tarball have a modification time set to the release date.

This should be documented in our release process document, once we write it (hopefully, while we release 1.4).

An interesting observation is that svn export gives you the last-modified-in-SVN date, while svn checkout gives you "now".

aaugustin@aaugustin Desktop % svn checkout https://code.djangoproject.com/svn/django/trunk/django/template                           ~/Desktop
(... snip ...)
Checked out revision 17200.
aaugustin@aaugustin Desktop % ls -al template                                                                                        ~/Desktop
total 360
drwxr-xr-x  14 aaugustin  staff    476 12 déc 19:47 ./
drwx------+ 14 aaugustin  staff    476 12 déc 19:47 ../
drwxr-xr-x   9 aaugustin  staff    306 12 déc 19:47 .svn/
-rw-r--r--   1 aaugustin  staff   3247 12 déc 19:47 __init__.py
-rw-r--r--   1 aaugustin  staff  49210 12 déc 19:47 base.py
-rw-r--r--   1 aaugustin  staff   6146 12 déc 19:47 context.py
-rw-r--r--   1 aaugustin  staff   3586 12 déc 19:47 debug.py
-rw-r--r--   1 aaugustin  staff  28093 12 déc 19:47 defaultfilters.py
-rw-r--r--   1 aaugustin  staff  46886 12 déc 19:47 defaulttags.py
-rw-r--r--   1 aaugustin  staff   7886 12 déc 19:47 loader.py
-rw-r--r--   1 aaugustin  staff  10519 12 déc 19:47 loader_tags.py
drwxr-xr-x   8 aaugustin  staff    272 12 déc 19:47 loaders/
-rw-r--r--   1 aaugustin  staff   6205 12 déc 19:47 response.py
-rw-r--r--   1 aaugustin  staff   6245 12 déc 19:47 smartif.py
aaugustin@aaugustin Desktop % rm -rf template                                                                                        ~/Desktop
aaugustin@aaugustin Desktop % svn export https://code.djangoproject.com/svn/django/trunk/django/template                             ~/Desktop
(... snip ...)
Exported revision 17200.
aaugustin@aaugustin Desktop % ls -al template                                                                                        ~/Desktop
total 360
drwxr-xr-x  13 aaugustin  staff    442 12 déc 19:47 ./
drwx------+ 14 aaugustin  staff    476 12 déc 19:47 ../
-rw-r--r--   1 aaugustin  staff   3247 27 nov  2010 __init__.py
-rw-r--r--   1 aaugustin  staff  49210  9 déc 23:10 base.py
-rw-r--r--   1 aaugustin  staff   6146 18 nov 14:01 context.py
-rw-r--r--   1 aaugustin  staff   3586 18 nov 16:00 debug.py
-rw-r--r--   1 aaugustin  staff  28093 18 nov 14:01 defaultfilters.py
-rw-r--r--   1 aaugustin  staff  46886  9 déc 23:13 defaulttags.py
-rw-r--r--   1 aaugustin  staff   7886 21 sep 16:20 loader.py
-rw-r--r--   1 aaugustin  staff  10519  9 déc 23:10 loader_tags.py
drwxr-xr-x   7 aaugustin  staff    238 12 déc 19:47 loaders/
-rw-r--r--   1 aaugustin  staff   6205 30 sep 13:46 response.py
-rw-r--r--   1 aaugustin  staff   6245 13 jul 11:35 smartif.py
aaugustin@aaugustin Desktop % rm -rf template                                                                                        ~/Desktop

[Aymeric Augustin]

Marking RFC since this ticket is only of interest to core devs, there's nothing more the community can do.

[Luke Plant]

I'm marking this a release blocker, since it is quite possibly security related: you could think you have installed a fix, but haven't at all.

[Aymeric Augustin]

Unfortunately, we can only fix it in the release process, which (to the best of my knowledge) isn't documented, except in the release manager's brains...

[Jannis Leidel]

FWIW, this doesn't happen if you use pip to install Django, so I suppose we should do two things:

• touch the files before packaging them to reset the date

• recommend using pip to install the Django source distribution

[Christopher Medrela]

Recommending to use pip - draft.

[Christopher Medrela]

I've sent initial draft.

Some changes on download page are required.

[Aymeric Augustin]

Recommending to use pip rather than easy_install to install Django (and other libs) is already covered by #9995 (#16471 being a duplicate).

Both #9995 and the docs part of this ticket should be fixed in the same commit.

[Aymeric Augustin]

To clarify: my previous comment means that this ticket is only about "touch the files before packaging them to reset the date".

[Aymeric Augustin]

In [17636]:

(The changeset message doesn't reference this ticket)

[Claude Paroz]

Is any action still required to close this ticket?

[Jannis Leidel]

Not from what I can see.

[Harry Percival]

This warning still pops up when a user installs Django into a virtualenv setup with --system-site-packages, but I think it's inaccurate in that case?

​http://virtualenvdemo.pythonanywhere.com/#note