Opened 8 years ago

Closed 5 years ago

Last modified 3 years ago

#9112 closed Bug (fixed)

New releases do not reliably overwrite old ones

Reported by: holdenweb Owned by: packagers
Component: Documentation Version: 1.0
Severity: Release blocker Keywords: release engineering
Cc: krzysiumed@… Triage Stage: Ready for checkin
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

There have been several issues reported where .pyc files from older releases apparently outlive the installation of a newer release.

The reason for this is that currently releases appear to be cut without altering the modification date of distributed files. Thus a .py file has not been changed since Jan 12, 2007 will appear in the release with that modification date.

When the release is installed with distutils the new .py files will replace their older versions, but if the .pyc from the prior release was compiled *after* the new .py file was last modified in SVN the new file will appear older than its corresponding .pyc, and the compiled file from the prior release will be retained, to the consternation of the installing user.

The simplest fix for this would appear to be to change the release engineering process so that after extraction from SVN all source files are touched to bring their modification date close to that of the release.

Strictly speaking this might also be considered a distutils bug, but the issue has not been reported elsewhere.

Attachments (1)

ticket_9112.diff (1.4 KB) - added by Christopher Medrela 5 years ago.
Recommending to use pip - draft.

Download all attachments as: .zip

Change History (20)

comment:1 Changed 8 years ago by Karen Tracey

#9090 is one symptom of this problem. As described there, the problem is not that the .pyc files are not updated but that the newer .py files are not copied to the installation directory. So even clearing out all the .pyc files in the install tree after this has happened won't produce a consistent level, since you've got a mixture of old and new .py files installed. Also, more discussion is here: http://groups.google.com/group/django-developers/browse_thread/thread/85d9500f319369a1. The problem seems to be resulting from old release tarfiles using tarfile creation date for contained files while 1.0 has last-modified-in-svn dates. That makes some files in the 1.0 release appear to be older than what was previously installed. (I do not think there is a distutils bug here.)

comment:2 Changed 8 years ago by (none)

milestone: post-1.0

Milestone post-1.0 deleted

comment:3 Changed 8 years ago by Jacob

Triage Stage: UnreviewedDesign decision needed

So what's the fix here? Touch all the files as part of rolling the distribution (i.e. from setup.py sdist)?

comment:4 Changed 6 years ago by Luke Plant

Severity: Normal
Type: Bug

comment:5 Changed 5 years ago by Aymeric Augustin

Easy pickings: unset
Triage Stage: Design decision neededAccepted
UI/UX: unset

Given Karen's last message in the thread linked above, the safest solution is to ensure all files in the tarball have a modification time set to the release date.

This should be documented in our release process document, once we write it (hopefully, while we release 1.4).

An interesting observation is that svn export gives you the last-modified-in-SVN date, while svn checkout gives you "now".

aaugustin@aaugustin Desktop % svn checkout https://code.djangoproject.com/svn/django/trunk/django/template                           ~/Desktop
(... snip ...)
Checked out revision 17200.
aaugustin@aaugustin Desktop % ls -al template                                                                                        ~/Desktop
total 360
drwxr-xr-x  14 aaugustin  staff    476 12 déc 19:47 ./
drwx------+ 14 aaugustin  staff    476 12 déc 19:47 ../
drwxr-xr-x   9 aaugustin  staff    306 12 déc 19:47 .svn/
-rw-r--r--   1 aaugustin  staff   3247 12 déc 19:47 __init__.py
-rw-r--r--   1 aaugustin  staff  49210 12 déc 19:47 base.py
-rw-r--r--   1 aaugustin  staff   6146 12 déc 19:47 context.py
-rw-r--r--   1 aaugustin  staff   3586 12 déc 19:47 debug.py
-rw-r--r--   1 aaugustin  staff  28093 12 déc 19:47 defaultfilters.py
-rw-r--r--   1 aaugustin  staff  46886 12 déc 19:47 defaulttags.py
-rw-r--r--   1 aaugustin  staff   7886 12 déc 19:47 loader.py
-rw-r--r--   1 aaugustin  staff  10519 12 déc 19:47 loader_tags.py
drwxr-xr-x   8 aaugustin  staff    272 12 déc 19:47 loaders/
-rw-r--r--   1 aaugustin  staff   6205 12 déc 19:47 response.py
-rw-r--r--   1 aaugustin  staff   6245 12 déc 19:47 smartif.py
aaugustin@aaugustin Desktop % rm -rf template                                                                                        ~/Desktop
aaugustin@aaugustin Desktop % svn export https://code.djangoproject.com/svn/django/trunk/django/template                             ~/Desktop
(... snip ...)
Exported revision 17200.
aaugustin@aaugustin Desktop % ls -al template                                                                                        ~/Desktop
total 360
drwxr-xr-x  13 aaugustin  staff    442 12 déc 19:47 ./
drwx------+ 14 aaugustin  staff    476 12 déc 19:47 ../
-rw-r--r--   1 aaugustin  staff   3247 27 nov  2010 __init__.py
-rw-r--r--   1 aaugustin  staff  49210  9 déc 23:10 base.py
-rw-r--r--   1 aaugustin  staff   6146 18 nov 14:01 context.py
-rw-r--r--   1 aaugustin  staff   3586 18 nov 16:00 debug.py
-rw-r--r--   1 aaugustin  staff  28093 18 nov 14:01 defaultfilters.py
-rw-r--r--   1 aaugustin  staff  46886  9 déc 23:13 defaulttags.py
-rw-r--r--   1 aaugustin  staff   7886 21 sep 16:20 loader.py
-rw-r--r--   1 aaugustin  staff  10519  9 déc 23:10 loader_tags.py
drwxr-xr-x   7 aaugustin  staff    238 12 déc 19:47 loaders/
-rw-r--r--   1 aaugustin  staff   6205 30 sep 13:46 response.py
-rw-r--r--   1 aaugustin  staff   6245 13 jul 11:35 smartif.py
aaugustin@aaugustin Desktop % rm -rf template                                                                                        ~/Desktop

comment:6 Changed 5 years ago by Aymeric Augustin

Triage Stage: AcceptedReady for checkin

Marking RFC since this ticket is only of interest to core devs, there's nothing more the community can do.

comment:7 Changed 5 years ago by Luke Plant

Severity: NormalRelease blocker

I'm marking this a release blocker, since it is quite possibly security related: you could think you have installed a fix, but haven't at all.

comment:8 Changed 5 years ago by Aymeric Augustin

Unfortunately, we can only fix it in the release process, which (to the best of my knowledge) isn't documented, except in the release manager's brains...

comment:9 Changed 5 years ago by Jannis Leidel

Component: Core (Other)Documentation

FWIW, this doesn't happen if you use pip to install Django, so I suppose we should do two things:

  • touch the files before packaging them to reset the date
  • recommend using pip to install the Django source distribution

comment:10 Changed 5 years ago by Jannis Leidel

Triage Stage: Ready for checkinAccepted

Changed 5 years ago by Christopher Medrela

Attachment: ticket_9112.diff added

Recommending to use pip - draft.

comment:11 Changed 5 years ago by Christopher Medrela

Cc: krzysiumed@… added
Has patch: set

I've sent initial draft.

Some changes on download page are required.

comment:12 Changed 5 years ago by Aymeric Augustin

Recommending to use pip rather than easy_install to install Django (and other libs) is already covered by #9995 (#16471 being a duplicate).

Both #9995 and the docs part of this ticket should be fixed in the same commit.

comment:13 Changed 5 years ago by Aymeric Augustin

Owner: changed from nobody to Aymeric Augustin

comment:14 Changed 5 years ago by Aymeric Augustin

To clarify: my previous comment means that this ticket is only about "touch the files before packaging them to reset the date".

comment:15 Changed 5 years ago by Aymeric Augustin

In [17636]:

(The changeset message doesn't reference this ticket)

comment:16 Changed 5 years ago by Aymeric Augustin

Owner: changed from Aymeric Augustin to packagers
Triage Stage: AcceptedReady for checkin

comment:17 Changed 5 years ago by Claude Paroz

Is any action still required to close this ticket?

comment:18 Changed 5 years ago by Jannis Leidel

Resolution: fixed
Status: newclosed

Not from what I can see.

comment:19 Changed 3 years ago by Harry Percival

This warning still pops up when a user installs Django into a virtualenv setup with --system-site-packages, but I think it's inaccurate in that case?

http://virtualenvdemo.pythonanywhere.com/#note

Note: See TracTickets for help on using tickets.
Back to Top