Models with datetime fields fail to save when date is below 1900 and USE_TZ is True

[Alexey Boriskin]

I have model with datetime field in it. When I save this model in admin site and the filled date is below 1900, I am getting the following traceback:

  File "/Users/voidus/Documents/workspace/django-trunk/django/contrib/admin/options.py", line 709, in save_model
    obj.save()
  File "/Users/voidus/Documents/workspace/django-trunk/django/db/models/base.py", line 524, in save
    force_update=force_update, update_fields=update_fields)
  File "/Users/voidus/Documents/workspace/django-trunk/django/db/models/base.py", line 595, in save_base
    rows = manager.using(using).filter(pk=pk_val)._update(values)
  File "/Users/voidus/Documents/workspace/django-trunk/django/db/models/query.py", line 573, in _update
    return query.get_compiler(self.db).execute_sql(None)
  File "/Users/voidus/Documents/workspace/django-trunk/django/db/models/sql/compiler.py", line 991, in execute_sql
    cursor = super(SQLUpdateCompiler, self).execute_sql(result_type)
  File "/Users/voidus/Documents/workspace/django-trunk/django/db/models/sql/compiler.py", line 813, in execute_sql
    sql, params = self.as_sql()
  File "/Users/voidus/Documents/workspace/django-trunk/django/db/models/sql/compiler.py", line 956, in as_sql
    val = field.get_db_prep_save(val, connection=self.connection)
  File "/Users/voidus/Documents/workspace/django-trunk/django/db/models/fields/__init__.py", line 303, in get_db_prep_save
    prepared=False)
  File "/Users/voidus/Documents/workspace/django-trunk/django/db/models/fields/__init__.py", line 834, in get_db_prep_value
    value = self.get_prep_value(value)
  File "/Users/voidus/Documents/workspace/django-trunk/django/db/models/fields/__init__.py", line 820, in get_prep_value
    if value is not None and settings.USE_TZ and timezone.is_naive(value):
  File "/Users/voidus/Documents/workspace/django-trunk/django/utils/timezone.py", line 271, in is_naive
    return value.tzinfo is None or value.tzinfo.utcoffset(value) is None
  File "/Users/voidus/Documents/workspace/django-trunk/django/utils/timezone.py", line 71, in utcoffset
    if self._isdst(dt):
  File "/Users/voidus/Documents/workspace/django-trunk/django/utils/timezone.py", line 89, in _isdst
    stamp = _time.mktime(tt)
ValueError: year out of range

I think that is because utcoffset and some other functions use _isdst function, which calls time.mktime and it have some limitation on the date range it can handle.

time module can handle dates in the more narrow range (1969-2038) then datetime module (1-9999).

[Alexey Boriskin]

As ​wikipedia article says, DST was not used before at least 1916. So, maybe simple check like

if dt.year < 1916:
    return False

will work

[Alexey Boriskin]

The other corner case is dates after 2038. While simple check for < 1916 will work for dates before 1900, as there were no DST at that time, after 2038 we will probably still have DST. 2038 is not near, but I can imagine a calendar application with events in future.

[Aymeric Augustin]

Indeed, this is a limitation of django.utils.tzinfo.LocalTimezone.

Could you try installing pytz (as strongly recommeded by the docs)? If that fixes the problem, I'd lean towards a documentation fix rather than extending the huge hack that is LocalTimezone.

Note that dates after 2038 are handled by returning incorrect data rather than crashing. I deeply dislike that, but I can't change it easily, for backwards-compatibility reasons.

[Aymeric Augustin]

See also #17262.

[Aymeric Augustin]

Waiting to know if installing pytz resolves the problem...

[anonymous]

Yes, installing pytz resolves the problem.

[Alexey Boriskin]

Sorry, that was me, was logged out somehow.

[Aymeric Augustin]

I'm accepting the ticket primarily as a documentation issue. This is related to #17262. I still have to decide how to deal with this.

[Alexey Boriskin]

Maybe it would be OK to require pytz installed if user set USE_I18N = True in settings.py?
Or maybe we can even set a hard dependency? I think it's now commonly consented that django should make more use of pip - at least localflavor will be separated, etc. So, maybe pytz would be the first thing to require?

[Aymeric Augustin]

USE_TZ = True is in the default settings files (for a good reason) and we want to keep Django usable without having to learn pip / virtualenv first (also for a good reason). I went to great lengths to ensure that pytz wouldn't be a mandatory.

I think it's more promising to raise a more explicit error in this case, ie. "value outside of supported range, please install pytz".

I

[Alexey Boriskin]

Yeah, it may be a good solution. But may this behaviour be considered as a regression? Imagine, you have some values outside of supported range in database, then upgrade django one version upper, set USE_TZ=True, as docs insist. And live being happy until someday you see in logs such message about pytz.

[Matthew Somerville]

Just to back up void's comment, I have just upgraded a site from Django 1.1 to 1.5. One of the few things that only bit me once it went live (and thanks as ever for the good documentation on upgrades and consistency :) ) was that I have some very old datetimes in the live database (back to the 17th century) that were working fine, that then started to throw "ValueError: year out of range" 500s due to this issue. It does indeed resolve if pytz is installed (or if USE_TZ is set back to False, which I've done for now).

[Aymeric Augustin <aymeric.augustin@…>]

In ded11aa620e5b38c78ff92e691a459d1977a2539:

Fixed #18766 -- Pointed to pytz when LocalTimezone fails.

Thanks void for the report.

[Aymeric Augustin <aymeric.augustin@…>]

In 7c31e195db963ad1ebb28265e721097000fa84d1:

[1.6.x] Fixed #18766 -- Pointed to pytz when LocalTimezone fails.

Thanks void for the report.

Backport of ded11aa6 from master.

[Aymeric Augustin <aymeric.augustin@…>]

In 1a1e14786aa35b4d80b46be747b5128800cf0f97:

Hardened the test introduced in ded11aa6. Refs #18766.

Inputs acceptable to time.mktime are platform-dependent.

[Aymeric Augustin <aymeric.augustin@…>]

In 0035a0ce2ec243ff9255157117c20caf41a61284:

[1.6.x] Hardened the test introduced in ded11aa6. Refs #18766.

Inputs acceptable to time.mktime are platform-dependent.

Backport of 1a1e1478 from master.

[Aymeric Augustin <aymeric.augustin@…>]

In c687bf0620d67e0579302515904ffe169581d036:

Further hardening. Refs #18766.

[Aymeric Augustin <aymeric.augustin@…>]

In b085e7c3033d49e4d7525bacb1f34340730918a8:

[1.6.x] Further hardening. Refs #18766.

Backport of c687bf0 from master.