Opened 18 years ago

Closed 16 years ago

Last modified 6 years ago

#1443 closed defect (fixed)

No support for dates before 1900

Reported by: Antti Kaihola Owned by: Adrian Holovaty
Component: Core (Other) Version: dev
Severity: normal Keywords: date 1900 datetime
Cc: rafaelbm_sbo@…, gary.wilson@…, brian.mabry.edwards@…, greenwaldjared@…, simon@… Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

See logviewer for a short discussion.

#1070 changed date validation so dates before 1900 raise an error. But that's only to avoid problems further down the road.

>>> datetime(1805,12,24).strftime('%Y-%m-%d')
ValueError: year=1805 is before 1900; the datetime strftime() methods require year >= 1900

Using PHP-style date formatting is broken too:

>>> dateformat.format(datetime(1899,4,12), 'Y-m-d')
/usr/local/lib/djangomr2458/django/utils/tzinfo.py in _isdst(self, dt)
---> 50         stamp = time.mktime(tt)
ValueError: year out of range

The problem is deeper in Python (or C, or the OS, or whatever):

>>> time.mktime((1901,12,14,0,0,0,0,0,0))
-2147477992.0
>>> time.mktime((1901,12,13,0,0,0,0,0,0))
OverflowError: mktime argument out of range
>>> time.mktime((1899,12,31,0,0,0,0,0,0))
ValueError: year out of range

Would mx.DateTime be a solution?

>>> mx.DateTime.Date(1543,12,24).strftime('%Y-%m-%d')
'1543-12-24'

I realize that adding another external module dependency is not a Good Thing. Maybe it's worth checking if mx.DateTime is similar enough to be used as a drop-in replacement for datetime in case it's installed on the system? This way those who need to go back to 19th century or before could just install mx.DateTime and enjoy.

Attachments (6)

datetime_pre_1900.patch (17.0 KB ) - added by Chris Beaven 17 years ago.
patch for handling of pre-1900 strftime (with tests)
datetime_pre_1900.2.patch (17.1 KB ) - added by Chris Beaven 17 years ago.
updated for recent change to date validator
datetime_pre_1900.2.2.patch (17.1 KB ) - added by brian.mabry.edwards@… 17 years ago.
update patch for oldforms
date1900.diff (4.2 KB ) - added by jbronn 16 years ago.
datetime patch from Chris Green
1443_updated.diff (4.2 KB ) - added by Philippe Raoult 16 years ago.
updated patch against [7322]
datetime_safe.diff (13.6 KB ) - added by Chris Beaven 16 years ago.
New and improved!

Download all attachments as: .zip

Change History (35)

comment:1 by Adrian Holovaty, 18 years ago

Resolution: wontfix
Status: newclosed

I'm marking this as a wontfix for now, because adding support for mx.DateTime would introduce too much complexity.

comment:2 by Gary Wilson <gary.wilson@…>, 17 years ago

Cc: gary.wilson@… added
Resolution: wontfix
Status: closedreopened
Version: SVN

I don't think that we necessarily need to use mx.DateTime. At least we could fix the admin interface by not using strftime when saving values to the database. datetime.datetime has no problem creating dates before 1900.

by Chris Beaven, 17 years ago

Attachment: datetime_pre_1900.patch added

patch for handling of pre-1900 strftime (with tests)

comment:3 by Chris Beaven, 17 years ago

Summary: no support for dates before 1900[patch] no support for dates before 1900

by Chris Beaven, 17 years ago

Attachment: datetime_pre_1900.2.patch added

updated for recent change to date validator

comment:4 by Simon G. <dev@…>, 17 years ago

Keywords: date 1900 datetime added
Triage Stage: UnreviewedAccepted

Marked as Accepted as I do think dates prior to 1900 will be an issue in a non-trivial proportion of applications. It's unfortunate that Python drops the ball here, and this adds another layer of complexity but the version here by SmileyChris doesn't require the mx.DateTime addition, and appears to handle everything.

Some issues prior to check in though:
Are there any potential backwards compat. issues here?
Can we directly replace datetime with datetime_pg (& there are datetime calls all over the code)?
Is this going to be a performance issue?
Alternatively - is it possible to get users who need these date ranges to install mx.DateTime (e.g. use the python version by default, and override with this/mx.DateTime on a user-setting basis)?

comment:5 by anonymous, 17 years ago

Cc: brian.mabry.edwards@… added

by brian.mabry.edwards@…, 17 years ago

Attachment: datetime_pre_1900.2.2.patch added

update patch for oldforms

comment:6 by anonymous, 17 years ago

Cc: rafaelbm_sbo@… added

I'm also having problems with dates before 1900, using the current SVN version and newforms:

Traceback (most recent call last):
File "C:\Python25\lib\site-packages\django\core\handlers\base.py" in get_response
  77. response = callback(request, *callback_args, **callback_kwargs)
File "C:\Documents and Settings\Usu�rio\Desktop\projeto1\exemplos\projeto1\cadastro\views.py" in lista_cliente
  37. form.save()
File "C:\Python25\lib\site-packages\django\newforms\models.py" in save
  56. return save_instance(self, instance, fields, fail_message, commit)
File "C:\Python25\lib\site-packages\django\newforms\models.py" in save_instance
  35. instance.save()
File "C:\Python25\lib\site-packages\django\db\models\base.py" in save
  217. db_values = [f.get_db_prep_save(f.pre_save(self, False)) for f in non_pks]
File "C:\Python25\lib\site-packages\django\db\models\fields\__init__.py" in get_db_prep_save
  496. value = value.strftime('%Y-%m-%d')

  ValueError at /cadastro/clientes/8/
  year=1000 is before 1900; the datetime strftime() methods require year >= 1900 
Last edited 6 years ago by Tim Graham (previous) (diff)

comment:7 by Fredrik Lundh <fredrik@…>, 17 years ago

Since you're just overriding a couple of (broken) portions of the core implementation, I don't really see any performance or compatibility issues here. But that's me.

comment:8 by Fredrik Lundh <fredrik@…>, 17 years ago

On the other hand, it might be better to just put the robust strptime/strftime implementations in a support library, and make sure that Django uses those for all datetime formatting. I'll check with the sprint IRC folks.

comment:9 by James Bennett, 17 years ago

#3306 is a duplicate but also has a patch.

comment:10 by Fredrik Lundh <fredrik@…>, 17 years ago

The #3306 approach is more of a hack, though; it just fakes its way around the C time limitations, in order to support 1900-9999. But that doesn't help; I have dates between 1500 and 1900 in my databases... :-/

No time to dig deeper into this right now, but I'm now convinced that putting Dalke's code in a support library, and making sure to use that whenever the Django core converts dates, is the right solution. Introducing a Django-specific datetime type/module may be elegant, but it's also rather fragile if you have a view that gets data from non-Django sources, and want to render it through Django.

I can check the license with Andrew and adapt Chris' patch, if there's support for that approach.

comment:11 by Jeremy Dunck, 16 years ago

@Fredrik: +1

in reply to:  10 ; comment:12 by jbronn, 16 years ago

@Fredrik: +1 (I need this support as well)

in reply to:  12 comment:13 by Gijs Nijholt <gijs.nijholt@…>, 16 years ago

Replying to jbronn:

@Fredrik: +1 (I need this support as well)

me too.. I'd really like to see this fixed
years before 1900 is something that python/django should support by default imho
is this platform specific btw? (i've read that it's an issue on osx for example)

thanks

by jbronn, 16 years ago

Attachment: date1900.diff added

datetime patch from Chris Green

by Philippe Raoult, 16 years ago

Attachment: 1443_updated.diff added

updated patch against [7322]

comment:14 by Philippe Raoult, 16 years ago

Triage Stage: AcceptedReady for checkin

tests are working, we're all good.

comment:15 by Chris Beaven, 16 years ago

Resolution: fixed
Status: reopenedclosed
Triage Stage: Ready for checkinDesign decision needed

PhiR - I'm actually not that keen on that implementation. I can see some updates I could make to my original patch (wow, that was a long time ago) and still think it's a better way. Could you bring it up on django-dev please?

comment:16 by Chris Beaven, 16 years ago

Resolution: fixed
Status: closedreopened

(oops)

comment:17 by jetsaredim, 16 years ago

Cc: greenwaldjared@… added

comment:18 by jetsaredim, 16 years ago

Patch needs improvement: set

FYI, I tried to apply both the datetime_pre_1900.2.patch and the datetime_pre_1900.2.2.patch and both generate a bunch of rejected diffs. I tried to hand-patch the rejects from 1900.2, but when I go into the admin site and edit an entry to change the date to before 1900, I get a ValueError exception saying "year=1786 is before 1900; the datetime strftime() methods require year >= 1900". I'm guessing that its just really hard to patch a moving target and that trunk has just progressed too far from when these patches were generated to be valid now.

I'm not terribly knowledgeable about the internals of django beyond the basics. I don't really have the time to study the code to get the proper expertise needed to truly patch this into current trunk (as it is this simple site I'm working on is taking up most of what little free time I have). I'm certainly willing to apply whatever patch someone can generate (SmileyChris or whoever) and report back my findings. My code is not terribly sophisticated and I'm really just needing this to be able to store basic date information for some historical data (cemetery records, actually), so if there are specific test that people want performed, that information would be good to know about as well.

Thanks in advance.

comment:19 by jetsaredim, 16 years ago

Also, just to be clear. I didn't apply both patches to the same tree (though I'm not entirely sure that would be bad since 1900.2.2 is supposed to just additionally fix oldforms, but I digress...). I re-checked-out trunk between applying 1900.2.2 and then subsequently applying 1900.2 and then proceeding to attempt to hand-patch.

Sorry for the spam, but just wanted to clarify that.

by Chris Beaven, 16 years ago

Attachment: datetime_safe.diff added

New and improved!

comment:20 by Chris Beaven, 16 years ago

Patch needs improvement: unset
Summary: [patch] no support for dates before 1900No support for dates before 1900

Ok, so my naming sucked - the new module is called datetime_safe. The methods that you'll normally use are just datetime_safe.new_date and datetime_safe.new_datetime.

It should handle all form (old and new) and database saving fine. Django only uses a datetime_safe object when it needs to, so if you're doing your own strftime calls on pre-1900 dates, you'll need to convert the object yourself.

comment:21 by Chris Beaven, 16 years ago

Owner: changed from nobody to Chris Beaven
Status: reopenednew

comment:22 by jetsaredim, 16 years ago

Just tried the new patch. Applied cleanly to current trunk (rev 7407).

I was able to change the dates in my objects to pre-1900 dates via the admin interface without error. I was also able to retrieve the data via my previously-existing (generic) view code - think object_detail - without error.

If anyone can come up with other tests that should be performed, I'll oblige whatever I can.

comment:23 by Simon Litchfield <simon@…>, 16 years ago

Cc: simon@… added

The way newforms treats < 1900 dates, with or without mx.DateTime, is another consideration for this ticket.

See also #7014 for a patch with tests to raise ValidationError instead of ValueError for dates before 1900 on newforms DateField/DateTimeFields.

comment:24 by dan <dan@…>, 16 years ago

Cc: dan@… added

comment:25 by dan, 16 years ago

patch no longer cleanly applies (looks like a whitespace issue), and it fails unit tests:

Error while importing datetime_safe:  File "./runtests.py", line 130, in django_tests
    mod = load_app(model_label)
  File "/Users/dan/src/django/tests/django/db/models/loading.py", line 72, in load_app
    mod = __import__(app_name, {}, {}, ['models'])
ImportError: No module named datetime_safe

would it help for me to reroll this, or does SmileyChris have this under control?

comment:26 by Chris Beaven, 16 years ago

Feel free to re-roll it.

comment:27 by Adrian Holovaty, 16 years ago

Owner: changed from Chris Beaven to Adrian Holovaty
Status: newassigned
Triage Stage: Design decision neededAccepted

comment:28 by dan, 16 years ago

Cc: dan@… removed

comment:29 by Adrian Holovaty, 16 years ago

Resolution: fixed
Status: assignedclosed

(In [7946]) Fixed #1443 -- Django's various bits now support dates before 1900. Thanks to SmileyChris, Chris Green, Fredrik Lundh and others for patches and design help

Note: See TracTickets for help on using tickets.
Back to Top