Opened 9 years ago

Closed 7 years ago

#1443 closed defect (fixed)

No support for dates before 1900

Reported by: akaihola Owned by: adrian
Component: Core (Other) Version: master
Severity: normal Keywords: date 1900 datetime
Cc: rafaelbm_sbo@…, gary.wilson@…, brian.mabry.edwards@…, greenwaldjared@…, simon@… Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: UI/UX:

Description

See logviewer for a short discussion.

#1070 changed date validation so dates before 1900 raise an error. But that's only to avoid problems further down the road.

>>> datetime(1805,12,24).strftime('%Y-%m-%d')
ValueError: year=1805 is before 1900; the datetime strftime() methods require year >= 1900

Using PHP-style date formatting is broken too:

>>> dateformat.format(datetime(1899,4,12), 'Y-m-d')
/usr/local/lib/djangomr2458/django/utils/tzinfo.py in _isdst(self, dt)
---> 50         stamp = time.mktime(tt)
ValueError: year out of range

The problem is deeper in Python (or C, or the OS, or whatever):

>>> time.mktime((1901,12,14,0,0,0,0,0,0))
-2147477992.0
>>> time.mktime((1901,12,13,0,0,0,0,0,0))
OverflowError: mktime argument out of range
>>> time.mktime((1899,12,31,0,0,0,0,0,0))
ValueError: year out of range

Would mx.DateTime be a solution?

>>> mx.DateTime.Date(1543,12,24).strftime('%Y-%m-%d')
'1543-12-24'

I realize that adding another external module dependency is not a Good Thing. Maybe it's worth checking if mx.DateTime is similar enough to be used as a drop-in replacement for datetime in case it's installed on the system? This way those who need to go back to 19th century or before could just install mx.DateTime and enjoy.

Attachments (6)

datetime_pre_1900.patch (17.0 KB) - added by SmileyChris 9 years ago.
patch for handling of pre-1900 strftime (with tests)
datetime_pre_1900.2.patch (17.1 KB) - added by SmileyChris 9 years ago.
updated for recent change to date validator
datetime_pre_1900.2.2.patch (17.1 KB) - added by brian.mabry.edwards@… 8 years ago.
update patch for oldforms
date1900.diff (4.2 KB) - added by jbronn 8 years ago.
datetime patch from Chris Green
1443_updated.diff (4.2 KB) - added by PhiR 7 years ago.
updated patch against [7322]
datetime_safe.diff (13.6 KB) - added by SmileyChris 7 years ago.
New and improved!

Download all attachments as: .zip

Change History (35)

comment:1 Changed 9 years ago by adrian

  • Resolution set to wontfix
  • Status changed from new to closed

I'm marking this as a wontfix for now, because adding support for mx.DateTime would introduce too much complexity.

comment:2 Changed 9 years ago by Gary Wilson <gary.wilson@…>

  • Cc gary.wilson@… added
  • Resolution wontfix deleted
  • Status changed from closed to reopened
  • Version set to SVN

I don't think that we necessarily need to use mx.DateTime. At least we could fix the admin interface by not using strftime when saving values to the database. datetime.datetime has no problem creating dates before 1900.

Changed 9 years ago by SmileyChris

patch for handling of pre-1900 strftime (with tests)

comment:3 Changed 9 years ago by SmileyChris

  • Summary changed from no support for dates before 1900 to [patch] no support for dates before 1900

Changed 9 years ago by SmileyChris

updated for recent change to date validator

comment:4 Changed 9 years ago by Simon G. <dev@…>

  • Keywords date 1900 datetime added
  • Triage Stage changed from Unreviewed to Accepted

Marked as Accepted as I do think dates prior to 1900 will be an issue in a non-trivial proportion of applications. It's unfortunate that Python drops the ball here, and this adds another layer of complexity but the version here by SmileyChris doesn't require the mx.DateTime addition, and appears to handle everything.

Some issues prior to check in though:
Are there any potential backwards compat. issues here?
Can we directly replace datetime with datetime_pg (& there are datetime calls all over the code)?
Is this going to be a performance issue?
Alternatively - is it possible to get users who need these date ranges to install mx.DateTime (e.g. use the python version by default, and override with this/mx.DateTime on a user-setting basis)?

comment:5 Changed 8 years ago by anonymous

  • Cc brian.mabry.edwards@… added

Changed 8 years ago by brian.mabry.edwards@…

update patch for oldforms

comment:6 Changed 8 years ago by anonymous

  • Cc rafaelbm_sbo@… added

I'm also having problems with dates before 1900, using the current SVN version and newforms:

Traceback (most recent call last):
File "C:\Python25\lib\site-packages\django\core\handlers\base.py" in get_response

  1. response = callback(request, *callback_args, callback_kwargs)

File "C:\Documents and Settings\Usu�rio\Desktop\projeto1\exemplos\projeto1\cadastro\views.py" in lista_cliente

  1. form.save()

File "C:\Python25\lib\site-packages\django\newforms\models.py" in save

  1. return save_instance(self, instance, fields, fail_message, commit)

File "C:\Python25\lib\site-packages\django\newforms\models.py" in save_instance

  1. instance.save()

File "C:\Python25\lib\site-packages\django\db\models\base.py" in save

  1. db_values = [f.get_db_prep_save(f.pre_save(self, False)) for f in non_pks]

File "C:\Python25\lib\site-packages\django\db\models\fields\init.py" in get_db_prep_save

  1. value = value.strftime('%Y-%m-%d')

ValueError at /cadastro/clientes/8/
year=1000 is before 1900; the datetime strftime() methods require year >= 1900

comment:7 Changed 8 years ago by Fredrik Lundh <fredrik@…>

Since you're just overriding a couple of (broken) portions of the core implementation, I don't really see any performance or compatibility issues here. But that's me.

comment:8 Changed 8 years ago by Fredrik Lundh <fredrik@…>

On the other hand, it might be better to just put the robust strptime/strftime implementations in a support library, and make sure that Django uses those for all datetime formatting. I'll check with the sprint IRC folks.

comment:9 Changed 8 years ago by ubernostrum

#3306 is a duplicate but also has a patch.

comment:10 follow-up: Changed 8 years ago by Fredrik Lundh <fredrik@…>

The #3306 approach is more of a hack, though; it just fakes its way around the C time limitations, in order to support 1900-9999. But that doesn't help; I have dates between 1500 and 1900 in my databases... :-/

No time to dig deeper into this right now, but I'm now convinced that putting Dalke's code in a support library, and making sure to use that whenever the Django core converts dates, is the right solution. Introducing a Django-specific datetime type/module may be elegant, but it's also rather fragile if you have a view that gets data from non-Django sources, and want to render it through Django.

I can check the license with Andrew and adapt Chris' patch, if there's support for that approach.

comment:11 Changed 8 years ago by jdunck

@Fredrik: +1

comment:12 in reply to: ↑ 10 ; follow-up: Changed 8 years ago by jbronn

@Fredrik: +1 (I need this support as well)

comment:13 in reply to: ↑ 12 Changed 8 years ago by Gijs Nijholt <gijs.nijholt@…>

Replying to jbronn:

@Fredrik: +1 (I need this support as well)

me too.. I'd really like to see this fixed
years before 1900 is something that python/django should support by default imho
is this platform specific btw? (i've read that it's an issue on osx for example)

thanks

Changed 8 years ago by jbronn

datetime patch from Chris Green

Changed 7 years ago by PhiR

updated patch against [7322]

comment:14 Changed 7 years ago by PhiR

  • Triage Stage changed from Accepted to Ready for checkin

tests are working, we're all good.

comment:15 Changed 7 years ago by SmileyChris

  • Resolution set to fixed
  • Status changed from reopened to closed
  • Triage Stage changed from Ready for checkin to Design decision needed

PhiR - I'm actually not that keen on that implementation. I can see some updates I could make to my original patch (wow, that was a long time ago) and still think it's a better way. Could you bring it up on django-dev please?

comment:16 Changed 7 years ago by SmileyChris

  • Resolution fixed deleted
  • Status changed from closed to reopened

(oops)

comment:17 Changed 7 years ago by jetsaredim

  • Cc greenwaldjared@… added

comment:18 Changed 7 years ago by jetsaredim

  • Patch needs improvement set

FYI, I tried to apply both the datetime_pre_1900.2.patch and the datetime_pre_1900.2.2.patch and both generate a bunch of rejected diffs. I tried to hand-patch the rejects from 1900.2, but when I go into the admin site and edit an entry to change the date to before 1900, I get a ValueError exception saying "year=1786 is before 1900; the datetime strftime() methods require year >= 1900". I'm guessing that its just really hard to patch a moving target and that trunk has just progressed too far from when these patches were generated to be valid now.

I'm not terribly knowledgeable about the internals of django beyond the basics. I don't really have the time to study the code to get the proper expertise needed to truly patch this into current trunk (as it is this simple site I'm working on is taking up most of what little free time I have). I'm certainly willing to apply whatever patch someone can generate (SmileyChris or whoever) and report back my findings. My code is not terribly sophisticated and I'm really just needing this to be able to store basic date information for some historical data (cemetery records, actually), so if there are specific test that people want performed, that information would be good to know about as well.

Thanks in advance.

comment:19 Changed 7 years ago by jetsaredim

Also, just to be clear. I didn't apply both patches to the same tree (though I'm not entirely sure that would be bad since 1900.2.2 is supposed to just additionally fix oldforms, but I digress...). I re-checked-out trunk between applying 1900.2.2 and then subsequently applying 1900.2 and then proceeding to attempt to hand-patch.

Sorry for the spam, but just wanted to clarify that.

Changed 7 years ago by SmileyChris

New and improved!

comment:20 Changed 7 years ago by SmileyChris

  • Patch needs improvement unset
  • Summary changed from [patch] no support for dates before 1900 to No support for dates before 1900

Ok, so my naming sucked - the new module is called datetime_safe. The methods that you'll normally use are just datetime_safe.new_date and datetime_safe.new_datetime.

It should handle all form (old and new) and database saving fine. Django only uses a datetime_safe object when it needs to, so if you're doing your own strftime calls on pre-1900 dates, you'll need to convert the object yourself.

comment:21 Changed 7 years ago by SmileyChris

  • Owner changed from nobody to SmileyChris
  • Status changed from reopened to new

comment:22 Changed 7 years ago by jetsaredim

Just tried the new patch. Applied cleanly to current trunk (rev 7407).

I was able to change the dates in my objects to pre-1900 dates via the admin interface without error. I was also able to retrieve the data via my previously-existing (generic) view code - think object_detail - without error.

If anyone can come up with other tests that should be performed, I'll oblige whatever I can.

comment:23 Changed 7 years ago by Simon Litchfield <simon@…>

  • Cc simon@… added

The way newforms treats < 1900 dates, with or without mx.DateTime, is another consideration for this ticket.

See also #7014 for a patch with tests to raise ValidationError instead of ValueError for dates before 1900 on newforms DateField/DateTimeFields.

comment:24 Changed 7 years ago by dan <dan@…>

  • Cc dan@… added

comment:25 Changed 7 years ago by dan90

patch no longer cleanly applies (looks like a whitespace issue), and it fails unit tests:

Error while importing datetime_safe:  File "./runtests.py", line 130, in django_tests
    mod = load_app(model_label)
  File "/Users/dan/src/django/tests/django/db/models/loading.py", line 72, in load_app
    mod = __import__(app_name, {}, {}, ['models'])
ImportError: No module named datetime_safe

would it help for me to reroll this, or does SmileyChris have this under control?

comment:26 Changed 7 years ago by SmileyChris

Feel free to re-roll it.

comment:27 Changed 7 years ago by adrian

  • Owner changed from SmileyChris to adrian
  • Status changed from new to assigned
  • Triage Stage changed from Design decision needed to Accepted

comment:28 Changed 7 years ago by dan90

  • Cc dan@… removed

comment:29 Changed 7 years ago by adrian

  • Resolution set to fixed
  • Status changed from assigned to closed

(In [7946]) Fixed #1443 -- Django's various bits now support dates before 1900. Thanks to SmileyChris, Chris Green, Fredrik Lundh and others for patches and design help

Note: See TracTickets for help on using tickets.
Back to Top