id	summary	reporter	owner	description	type	status	component	version	severity	resolution	keywords	cc	stage	has_patch	needs_docs	needs_tests	needs_better_patch	easy	ui_ux
37392	Docs for custom db_type() fields are misleading about get_internal_type() -- can silently break exact-match lookups on Django 5.0+	Robert Raposa		"== Background ==

This ticket was prompted by a real production incident: a Django 5.2 upgrade silently broke
updates to existing rows once a table's primary keys passed 2^31^-1, in a field type that had
followed the docs' current advice (see below). This draft was written with AI assistance
(Claude), disclosed here for transparency, consistent with Trac's practice of asking reporters to
note AI tool use (see discussion on #36484).

== Description ==

The ""Emulating built-in field types"" section of the custom-model-fields how-to
(https://docs.djangoproject.com/en/6.1/howto/custom-model-fields/#emulating-built-in-field-types)
opens with:

> If you have created a `db_type()` method, you don't need to worry about
> `get_internal_type()` -- it won't be used much.

This is backwards for any field that widens or changes the *range* of a built-in type via
`db_type()`/`rel_db_type()` without also updating `get_internal_type()`. Since Django 5.0
(unchanged through 5.1 and 5.2; added in dde2537fbb04, ""Fixed #27397 -- Prevented integer
overflows on integer field lookups""), exact-match lookups (`IntegerFieldExact`, via
`IntegerFieldOverflow` in `django/db/models/lookups.py`) call
`connection.ops.integer_field_range(field.get_internal_type())`
*before building any SQL*, and raise `EmptyResultSet` if the lookup value falls outside that
range -- silently returning zero rows, with no exception, no query sent to the database.
`get_internal_type()` is very much ""used,"" just not where this doc section implies.

Related: #28702 hit the same shape in 2017 for CIText fields (`db_type()` wasn't consulted where
the docs implied it would be, this time for casting rather than range). A comment there noted the
docs ""need[] to be adjusted"" -- but the fix landed only for that specific case; the generic
misleading sentence above is still present today.

== Reproduction ==

{{{#!python
from django.db import models
from django.db.models import AutoField

class UnsignedBigIntAutoField(AutoField):
    def db_type(self, connection):
        return 'bigint UNSIGNED AUTO_INCREMENT' if connection.vendor == 'mysql' else super().db_type(connection)
    def rel_db_type(self, connection):
        return 'bigint UNSIGNED' if connection.vendor == 'mysql' else super().rel_db_type(connection)

class Widget(models.Model):
    id = UnsignedBigIntAutoField(primary_key=True)
}}}

Following the doc's advice literally -- defining `db_type()`/`rel_db_type()` and not touching
`get_internal_type()` -- leaves `get_internal_type()` at the inherited `'AutoField'`. On MySQL,
once the table's AUTO_INCREMENT sequence passes 2^31^-1 (perfectly valid for an unsigned bigint
column, and not unusual for a long-lived high-traffic table):

{{{
>>> Widget.objects.filter(pk=4_312_695_480).exists()
False   # the row exists; this should be True
>>> Widget.objects.get(pk=4_312_695_480)
Widget.DoesNotExist
}}}

`Model.save(force_update=True)`'s internal `filter(pk=...).exists()` check hits the same path,
so this isn't just a read-lookup quirk: saves/updates to existing rows silently stop working
(raising `DatabaseError(""Forced update did not affect any rows."")` from Django's own save()
machinery) once a table's ids cross that boundary, with no application-level error pointing at
the real cause.

== Our fix ==

We hit this in production with a bigint-unsigned-PK AutoField subclass predating Django 5.0;
upgrading to Django 5.2 turned it into a live incident once ids passed 2^31^-1. Fix was a
one-line `get_internal_type()` override returning `'BigAutoField'` (wide enough 64-bit signed
range to never clip a real value, even though the column is unsigned):
edx/edx-platform#495.

== Possible follow-up (not proposing a patch here) ==

Django already has the plumbing to catch this class of bug at `manage.py check` time:
`Field._check_backend_specific_checks()` already calls
`connections[alias].validation.check_field()`, and MySQL's `check_field_type()`
(`django/db/backends/mysql/validation.py`) already receives both the field and its rendered
`db_type()`. Comparing that against `connection.data_types.get(field.get_internal_type())` and
warning on a mismatch could have caught this years before any table's ids reached 2^31^. Flagging
the idea for consideration, not attaching an implementation -- #36484 suggests new
settings/behavior changes get a high bar here, so happy to defer on whether/how this should be
pursued.
"	Bug	new	Documentation	5.0	Normal		custom fields, get_internal_type, db_type, IntegerFieldOverflow, AutoField		Unreviewed	0	0	0	0	0	0