Opened 7 years ago

Closed 3 years ago

Last modified 22 months ago

#9112 closed Bug (fixed)

New releases do not reliably overwrite old ones

Reported by: holdenweb Owned by: packagers
Component: Documentation Version: 1.0
Severity: Release blocker Keywords: release engineering
Cc: krzysiumed@… Triage Stage: Ready for checkin
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

There have been several issues reported where .pyc files from older releases apparently outlive the installation of a newer release.

The reason for this is that currently releases appear to be cut without altering the modification date of distributed files. Thus a .py file has not been changed since Jan 12, 2007 will appear in the release with that modification date.

When the release is installed with distutils the new .py files will replace their older versions, but if the .pyc from the prior release was compiled *after* the new .py file was last modified in SVN the new file will appear older than its corresponding .pyc, and the compiled file from the prior release will be retained, to the consternation of the installing user.

The simplest fix for this would appear to be to change the release engineering process so that after extraction from SVN all source files are touched to bring their modification date close to that of the release.

Strictly speaking this might also be considered a distutils bug, but the issue has not been reported elsewhere.

Attachments (1)

ticket_9112.diff (1.4 KB) - added by krzysiumed 3 years ago.
Recommending to use pip - draft.

Download all attachments as: .zip

Change History (20)

comment:1 Changed 7 years ago by kmtracey

  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset

#9090 is one symptom of this problem. As described there, the problem is not that the .pyc files are not updated but that the newer .py files are not copied to the installation directory. So even clearing out all the .pyc files in the install tree after this has happened won't produce a consistent level, since you've got a mixture of old and new .py files installed. Also, more discussion is here: http://groups.google.com/group/django-developers/browse_thread/thread/85d9500f319369a1. The problem seems to be resulting from old release tarfiles using tarfile creation date for contained files while 1.0 has last-modified-in-svn dates. That makes some files in the 1.0 release appear to be older than what was previously installed. (I do not think there is a distutils bug here.)

comment:2 Changed 6 years ago by anonymous

  • milestone post-1.0 deleted

Milestone post-1.0 deleted

comment:3 Changed 6 years ago by jacob

  • Triage Stage changed from Unreviewed to Design decision needed

So what's the fix here? Touch all the files as part of rolling the distribution (i.e. from setup.py sdist)?

comment:4 Changed 4 years ago by lukeplant

  • Severity set to Normal
  • Type set to Bug

comment:5 Changed 3 years ago by aaugustin

  • Easy pickings unset
  • Triage Stage changed from Design decision needed to Accepted
  • UI/UX unset

Given Karen's last message in the thread linked above, the safest solution is to ensure all files in the tarball have a modification time set to the release date.

This should be documented in our release process document, once we write it (hopefully, while we release 1.4).

An interesting observation is that svn export gives you the last-modified-in-SVN date, while svn checkout gives you "now".

aaugustin@aaugustin Desktop % svn checkout https://code.djangoproject.com/svn/django/trunk/django/template                           ~/Desktop
(... snip ...)
Checked out revision 17200.
aaugustin@aaugustin Desktop % ls -al template                                                                                        ~/Desktop
total 360
drwxr-xr-x  14 aaugustin  staff    476 12 déc 19:47 ./
drwx------+ 14 aaugustin  staff    476 12 déc 19:47 ../
drwxr-xr-x   9 aaugustin  staff    306 12 déc 19:47 .svn/
-rw-r--r--   1 aaugustin  staff   3247 12 déc 19:47 __init__.py
-rw-r--r--   1 aaugustin  staff  49210 12 déc 19:47 base.py
-rw-r--r--   1 aaugustin  staff   6146 12 déc 19:47 context.py
-rw-r--r--   1 aaugustin  staff   3586 12 déc 19:47 debug.py
-rw-r--r--   1 aaugustin  staff  28093 12 déc 19:47 defaultfilters.py
-rw-r--r--   1 aaugustin  staff  46886 12 déc 19:47 defaulttags.py
-rw-r--r--   1 aaugustin  staff   7886 12 déc 19:47 loader.py
-rw-r--r--   1 aaugustin  staff  10519 12 déc 19:47 loader_tags.py
drwxr-xr-x   8 aaugustin  staff    272 12 déc 19:47 loaders/
-rw-r--r--   1 aaugustin  staff   6205 12 déc 19:47 response.py
-rw-r--r--   1 aaugustin  staff   6245 12 déc 19:47 smartif.py
aaugustin@aaugustin Desktop % rm -rf template                                                                                        ~/Desktop
aaugustin@aaugustin Desktop % svn export https://code.djangoproject.com/svn/django/trunk/django/template                             ~/Desktop
(... snip ...)
Exported revision 17200.
aaugustin@aaugustin Desktop % ls -al template                                                                                        ~/Desktop
total 360
drwxr-xr-x  13 aaugustin  staff    442 12 déc 19:47 ./
drwx------+ 14 aaugustin  staff    476 12 déc 19:47 ../
-rw-r--r--   1 aaugustin  staff   3247 27 nov  2010 __init__.py
-rw-r--r--   1 aaugustin  staff  49210  9 déc 23:10 base.py
-rw-r--r--   1 aaugustin  staff   6146 18 nov 14:01 context.py
-rw-r--r--   1 aaugustin  staff   3586 18 nov 16:00 debug.py
-rw-r--r--   1 aaugustin  staff  28093 18 nov 14:01 defaultfilters.py
-rw-r--r--   1 aaugustin  staff  46886  9 déc 23:13 defaulttags.py
-rw-r--r--   1 aaugustin  staff   7886 21 sep 16:20 loader.py
-rw-r--r--   1 aaugustin  staff  10519  9 déc 23:10 loader_tags.py
drwxr-xr-x   7 aaugustin  staff    238 12 déc 19:47 loaders/
-rw-r--r--   1 aaugustin  staff   6205 30 sep 13:46 response.py
-rw-r--r--   1 aaugustin  staff   6245 13 jul 11:35 smartif.py
aaugustin@aaugustin Desktop % rm -rf template                                                                                        ~/Desktop

comment:6 Changed 3 years ago by aaugustin

  • Triage Stage changed from Accepted to Ready for checkin

Marking RFC since this ticket is only of interest to core devs, there's nothing more the community can do.

comment:7 Changed 3 years ago by lukeplant

  • Severity changed from Normal to Release blocker

I'm marking this a release blocker, since it is quite possibly security related: you could think you have installed a fix, but haven't at all.

comment:8 Changed 3 years ago by aaugustin

Unfortunately, we can only fix it in the release process, which (to the best of my knowledge) isn't documented, except in the release manager's brains...

comment:9 Changed 3 years ago by jezdez

  • Component changed from Core (Other) to Documentation

FWIW, this doesn't happen if you use pip to install Django, so I suppose we should do two things:

  • touch the files before packaging them to reset the date
  • recommend using pip to install the Django source distribution

comment:10 Changed 3 years ago by jezdez

  • Triage Stage changed from Ready for checkin to Accepted

Changed 3 years ago by krzysiumed

Recommending to use pip - draft.

comment:11 Changed 3 years ago by krzysiumed

  • Cc krzysiumed@… added
  • Has patch set

I've sent initial draft.

Some changes on download page are required.

comment:12 Changed 3 years ago by aaugustin

Recommending to use pip rather than easy_install to install Django (and other libs) is already covered by #9995 (#16471 being a duplicate).

Both #9995 and the docs part of this ticket should be fixed in the same commit.

comment:13 Changed 3 years ago by aaugustin

  • Owner changed from nobody to aaugustin

comment:14 Changed 3 years ago by aaugustin

To clarify: my previous comment means that this ticket is only about "touch the files before packaging them to reset the date".

comment:15 Changed 3 years ago by aaugustin

In [17636]:

(The changeset message doesn't reference this ticket)

comment:16 Changed 3 years ago by aaugustin

  • Owner changed from aaugustin to packagers
  • Triage Stage changed from Accepted to Ready for checkin

comment:17 Changed 3 years ago by claudep

Is any action still required to close this ticket?

comment:18 Changed 3 years ago by jezdez

  • Resolution set to fixed
  • Status changed from new to closed

Not from what I can see.

comment:19 Changed 22 months ago by hjwp

This warning still pops up when a user installs Django into a virtualenv setup with --system-site-packages, but I think it's inaccurate in that case?

http://virtualenvdemo.pythonanywhere.com/#note

Note: See TracTickets for help on using tickets.
Back to Top