Changes between Initial Version and Version 1 of Ticket #37392


Ignore:
Timestamp:
Oct 9, 2026, 6:09:46 AM (2 weeks ago)
Author:
Sarah Boyce
Comment:

Legend:

Unmodified
Added
Removed
Modified

  • Ticket #37392 – Description

    initial v1  
    11== Background ==
    22
    3 This ticket was prompted by a real production incident: a Django 5.2 upgrade silently broke
    4 updates to existing rows once a table's primary keys passed 2^31^-1, in a field type that had
    5 followed the docs' current advice (see below). This draft was written with AI assistance
    6 (Claude), disclosed here for transparency, consistent with Trac's practice of asking reporters to
    7 note AI tool use (see discussion on #36484).
     3This ticket was prompted by a real production incident: a Django 5.2 upgrade silently broke updates to existing rows once a table's primary keys passed 2^31^-1, in a field type that had followed the docs' current advice (see below).
     4
     5This draft was written with AI assistance (Claude), disclosed here for transparency, consistent with Trac's practice of asking reporters to note AI tool use (see discussion on #36484).
    86
    97== Description ==
    108
    11 The "Emulating built-in field types" section of the custom-model-fields how-to
    12 (https://docs.djangoproject.com/en/6.1/howto/custom-model-fields/#emulating-built-in-field-types)
    13 opens with:
     9The "Emulating built-in field types" section of the custom-model-fields how-to (https://docs.djangoproject.com/en/6.1/howto/custom-model-fields/#emulating-built-in-field-types) opens with:
    1410
    15 > If you have created a `db_type()` method, you don't need to worry about
    16 > `get_internal_type()` -- it won't be used much.
     11> If you have created a `db_type()` method, you don't need to worry about `get_internal_type()` -- it won't be used much.
    1712
    18 This is backwards for any field that widens or changes the *range* of a built-in type via
    19 `db_type()`/`rel_db_type()` without also updating `get_internal_type()`. Since Django 5.0
    20 (unchanged through 5.1 and 5.2; added in dde2537fbb04, "Fixed #27397 -- Prevented integer
    21 overflows on integer field lookups"), exact-match lookups (`IntegerFieldExact`, via
    22 `IntegerFieldOverflow` in `django/db/models/lookups.py`) call
    23 `connection.ops.integer_field_range(field.get_internal_type())`
    24 *before building any SQL*, and raise `EmptyResultSet` if the lookup value falls outside that
     13This is backwards for any field that widens or changes the *range* of a built-in type via `db_type()`/`rel_db_type()` without also updating `get_internal_type()`.
     14Since Django 5.0 (unchanged through 5.1 and 5.2; added in dde2537fbb04, "Fixed #27397 -- Prevented integer overflows on integer field lookups"), exact-match lookups (`IntegerFieldExact`, via `IntegerFieldOverflow` in `django/db/models/lookups.py`) call `connection.ops.integer_field_range(field.get_internal_type())` *before building any SQL*, and raise `EmptyResultSet` if the lookup value falls outside that
    2515range -- silently returning zero rows, with no exception, no query sent to the database.
     16
    2617`get_internal_type()` is very much "used," just not where this doc section implies.
    2718
    28 Related: #28702 hit the same shape in 2017 for CIText fields (`db_type()` wasn't consulted where
    29 the docs implied it would be, this time for casting rather than range). A comment there noted the
    30 docs "need[] to be adjusted" -- but the fix landed only for that specific case; the generic
    31 misleading sentence above is still present today.
     19Related: #28702 hit the same shape in 2017 for CIText fields (`db_type()` wasn't consulted where the docs implied it would be, this time for casting rather than range).
     20A comment there noted the docs "need[] to be adjusted" -- but the fix landed only for that specific case; the generic misleading sentence above is still present today.
    3221
    3322== Reproduction ==
    … …  
    4736}}}
    4837
    49 Following the doc's advice literally -- defining `db_type()`/`rel_db_type()` and not touching
    50 `get_internal_type()` -- leaves `get_internal_type()` at the inherited `'AutoField'`. On MySQL,
    51 once the table's AUTO_INCREMENT sequence passes 2^31^-1 (perfectly valid for an unsigned bigint
    52 column, and not unusual for a long-lived high-traffic table):
     38Following the doc's advice literally -- defining `db_type()`/`rel_db_type()` and not touching `get_internal_type()` -- leaves `get_internal_type()` at the inherited `'AutoField'`.
     39On MySQL, once the table's AUTO_INCREMENT sequence passes 2^31^-1 (perfectly valid for an unsigned bigint column, and not unusual for a long-lived high-traffic table):
    5340
    5441{{{
    … …  
    5946}}}
    6047
    61 `Model.save(force_update=True)`'s internal `filter(pk=...).exists()` check hits the same path,
    62 so this isn't just a read-lookup quirk: saves/updates to existing rows silently stop working
    63 (raising `DatabaseError("Forced update did not affect any rows.")` from Django's own save()
    64 machinery) once a table's ids cross that boundary, with no application-level error pointing at
    65 the real cause.
     48`Model.save(force_update=True)`'s internal `filter(pk=...).exists()` check hits the same path, so this isn't just a read-lookup quirk: saves/updates to existing rows silently stop working (raising `DatabaseError("Forced update did not affect any rows.")` from Django's own save() machinery) once a table's ids cross that boundary, with no application-level error pointing at the real cause.
    6649
    6750== Our fix ==
    6851
    69 We hit this in production with a bigint-unsigned-PK AutoField subclass predating Django 5.0;
    70 upgrading to Django 5.2 turned it into a live incident once ids passed 2^31^-1. Fix was a
    71 one-line `get_internal_type()` override returning `'BigAutoField'` (wide enough 64-bit signed
    72 range to never clip a real value, even though the column is unsigned):
    73 edx/edx-platform#495.
     52We hit this in production with a bigint-unsigned-PK AutoField subclass predating Django 5.0; upgrading to Django 5.2 turned it into a live incident once ids passed 2^31^-1.
     53Fix was a one-line `get_internal_type()` override returning `'BigAutoField'` (wide enough 64-bit signed range to never clip a real value, even though the column is unsigned): edx/edx-platform#495.
    7454
    7555== Possible follow-up (not proposing a patch here) ==
    7656
    7757Django already has the plumbing to catch this class of bug at `manage.py check` time:
    78 `Field._check_backend_specific_checks()` already calls
    79 `connections[alias].validation.check_field()`, and MySQL's `check_field_type()`
    80 (`django/db/backends/mysql/validation.py`) already receives both the field and its rendered
    81 `db_type()`. Comparing that against `connection.data_types.get(field.get_internal_type())` and
    82 warning on a mismatch could have caught this years before any table's ids reached 2^31^. Flagging
    83 the idea for consideration, not attaching an implementation -- #36484 suggests new
    84 settings/behavior changes get a high bar here, so happy to defer on whether/how this should be
    85 pursued.
     58`Field._check_backend_specific_checks()` already calls `connections[alias].validation.check_field()`, and MySQL's `check_field_type()` (`django/db/backends/mysql/validation.py`) already receives both the field and its rendered `db_type()`. Comparing that against `connection.data_types.get(field.get_internal_type())` and warning on a mismatch could have caught this years before any table's ids reached 2^31^. Flagging the idea for consideration, not attaching an implementation -- #36484 suggests new settings/behavior changes get a high bar here, so happy to defer on whether/how this should be pursued.
Back to Top