django.utils.timezone.make_naive can underflow for timezones close to datetime.min

[Liam DeVoe]

I'm a contributor over at ​Hypothesis. ​We found that saving (or retrieving?) models with aware DateTimeField values close to datetime.min can error if:

• timezone support is enabled (USE_TZ=True)
• you are using the sqlite database (presumably also any other db backend without tz aware support, if they exist?)

Explicitly:

from datetime import datetime, timezone
from zoneinfo import ZoneInfo
from django.utils.timezone import make_naive

dt = datetime.min.replace(tzinfo=ZoneInfo(key='Africa/Addis_Ababa'))
make_naive(dt, timezone.utc) # OverflowError

where make_naive is called from ​here. I would guess the overflow case for datetime.max is symmetric and similar as well.

I hope you'll forgive the lack of a django reproducer - I live in testing land and don't have a local django environment handy :). Hopefully the cause is clear and reproducible with vanilla django models.

[Sarah Boyce]

Replicated, thank you!

Should be an issue in MySQL and Oracle as well - refs #23820
Rough testcase:

tests/backends/sqlite/test_operations.py

@@ -1 +1 @@
 import unittest
+from datetime import datetime
 
 from django.core.management.color import no_style
 from django.db import connection
-from django.test import TestCase
+from django.test import TestCase, override_settings
+from django.utils import timezone
 
 from ..models import Person, Tag
 
@@ -86 +88 @@
             "zzz'",
             statements[-1],
         )
+
+    @override_settings(USE_TZ=True, TIME_ZONE="UTC")
+    def test_adapt_datetimefield_value_close_to_datetime_min(self):
+        africa_nairobi_tz = timezone.get_fixed_timezone(180)
+        value = datetime.min.replace(tzinfo=africa_nairobi_tz)
+
+        connection.ops.adapt_datetimefield_value(value)  # OverflowError

[Shubham Singh Sugara]

will try to work on it but will have to check first
Bug is causing issue on other db as well if so look for fix

[Tim Graham]

What behavior do you expect? If the timezone conversion causes the datetime to be too small or too big, what can be done?

[Shubham Singh Sugara]

Replying to Tim Graham:

What behavior do you expect? If the timezone conversion causes the datetime to be too small or too big, what can be done?

For now i can think of this solution
Max case should be also included if datetime.min is giving error than there is a possibility datetime.max also giving the error due to corner cases

We can Update make_naive Function:

Modify the make_naive function in Django to handle boundary cases(max and min). For instance, if an OverflowError is detected, the function could return the closest safe datetime value or handle the error gracefully in another way.

for now i think this can work across all db as changing other parts of modules will be db specific
but if any other suggestion open to discussion

[Tim Graham]

The use case for storing datetime.min and max is unclear, but it seems doubtful that we should silently change the value the user provided. I would close this ticket as "needs info" or "wontfix" unless the reporter can provide more explanation.

[Liam DeVoe]

I half expected this ticket to be wontfix when I opened it :). Here's one sane alternative I can think of: if you're converting to naive solely for serializing to the db, and when you deserialize from the db you make it tz aware again, then the only time it's out of bounds for the serialization/storage portion. But you control the serialization format, so you could extend the limits by 24 hours in each direction to accommodate those overflows, relying on the invariant that they will be made immediately tz aware and thus in bounds before reaching the user.

I'm not familiar with django internals, so this may not be acceptable or the assumed invariants may not be correct - that's up to you guys. I agree that silently capping is not a good idea. Possibly this should just be wontifx'd but documented; it does seem surprising to a user that I cannot store an in-bounds tz aware datetime, because there is no reason to expect it is converted to a naive datetime.

[Sarah Boyce]

Thank you both for the discussion!
Given I believe this was found and validated via testing scenarios (rather than "in real life"), and much of this code has been this way for years, going to update to wontfix.
If anyone finds that this causes them an issue with their application, they're welcome to add this discussion.