Clarify the role of get_internal_type() when overriding db_type()

[Robert Raposa]

Background

This ticket was prompted by a real production incident: a Django 5.2 upgrade silently broke updates to existing rows once a table's primary keys passed 2³¹-1, in a field type that had followed the docs' current advice (see below).

This draft was written with AI assistance (Claude), disclosed here for transparency, consistent with Trac's practice of asking reporters to note AI tool use (see discussion on #36484).

Description

The "Emulating built-in field types" section of the custom-model-fields how-to (​https://docs.djangoproject.com/en/6.1/howto/custom-model-fields/#emulating-built-in-field-types) opens with:

If you have created a db_type() method, you don't need to worry about get_internal_type() -- it won't be used much.

This is backwards for any field that widens or changes the *range* of a built-in type via db_type()/rel_db_type() without also updating get_internal_type().
Since Django 5.0 (unchanged through 5.1 and 5.2; added in dde2537fbb04, "Fixed #27397 -- Prevented integer overflows on integer field lookups"), exact-match lookups (IntegerFieldExact, via IntegerFieldOverflow in django/db/models/lookups.py) call connection.ops.integer_field_range(field.get_internal_type()) *before building any SQL*, and raise EmptyResultSet if the lookup value falls outside that
range -- silently returning zero rows, with no exception, no query sent to the database.

get_internal_type() is very much "used," just not where this doc section implies.

Related: #28702 hit the same shape in 2017 for CIText fields (db_type() wasn't consulted where the docs implied it would be, this time for casting rather than range).
A comment there noted the docs "need[] to be adjusted" -- but the fix landed only for that specific case; the generic misleading sentence above is still present today.

Reproduction

from django.db import models
from django.db.models import AutoField

class UnsignedBigIntAutoField(AutoField):
    def db_type(self, connection):
        return 'bigint UNSIGNED AUTO_INCREMENT' if connection.vendor == 'mysql' else super().db_type(connection)
    def rel_db_type(self, connection):
        return 'bigint UNSIGNED' if connection.vendor == 'mysql' else super().rel_db_type(connection)

class Widget(models.Model):
    id = UnsignedBigIntAutoField(primary_key=True)

Following the doc's advice literally -- defining db_type()/rel_db_type() and not touching get_internal_type() -- leaves get_internal_type() at the inherited 'AutoField'.
On MySQL, once the table's AUTO_INCREMENT sequence passes 2³¹-1 (perfectly valid for an unsigned bigint column, and not unusual for a long-lived high-traffic table):

>>> Widget.objects.filter(pk=4_312_695_480).exists()
False   # the row exists; this should be True
>>> Widget.objects.get(pk=4_312_695_480)
Widget.DoesNotExist

Model.save(force_update=True)'s internal filter(pk=...).exists() check hits the same path, so this isn't just a read-lookup quirk: saves/updates to existing rows silently stop working (raising DatabaseError("Forced update did not affect any rows.") from Django's own save() machinery) once a table's ids cross that boundary, with no application-level error pointing at the real cause.

Our fix

We hit this in production with a bigint-unsigned-PK AutoField subclass predating Django 5.0; upgrading to Django 5.2 turned it into a live incident once ids passed 2³¹-1.
Fix was a one-line get_internal_type() override returning 'BigAutoField' (wide enough 64-bit signed range to never clip a real value, even though the column is unsigned): edx/edx-platform#495.

Possible follow-up (not proposing a patch here)

Django already has the plumbing to catch this class of bug at manage.py check time:
Field._check_backend_specific_checks() already calls connections[alias].validation.check_field(), and MySQL's check_field_type() (django/db/backends/mysql/validation.py) already receives both the field and its rendered db_type(). Comparing that against connection.data_types.get(field.get_internal_type()) and warning on a mismatch could have caught this years before any table's ids reached 2³¹. Flagging the idea for consideration, not attaching an implementation -- #36484 suggests new settings/behavior changes get a high bar here, so happy to defer on whether/how this should be pursued.

[Sarah Boyce]

Thank you for the ticket
I agree we should update the docs, including the UnsignedAutoField example: ​https://docs.djangoproject.com/en/6.1/howto/custom-model-fields/#custom-database-types:~:text=UnsignedAutoField

I am not sure a system check is not going to also warn for false positives

[Vaibhav Pant]

Replying to Sarah Boyce:

Thank you for the ticket
I agree we should update the docs, including the UnsignedAutoField example: ​https://docs.djangoproject.com/en/6.1/howto/custom-model-fields/#custom-database-types:~:text=UnsignedAutoField

I am not sure a system check is not going to also warn for false positives

Thanks Sarah, I'd like to work on this one and will keep it docs-only, leaving the system check out.

One question about the UnsignedAutoField example: would you prefer it to include a get_internal_type() override returning "BigAutoField", as mentioned in the ticket, or should the example stay minimal, with the need for the override explained in the surrounding text? Happy to go with whichever you think fits best.

[Vaibhav Pant]

I've opened a PR for this: ​https://github.com/django/django/pull/22159